What Is Octo (Generalist Robot Policy)?

Octo is introduced in the paper Octo: An Open-Source Generalist Robot Policy (arXiv 2405.12213), with an official project page and GitHub repository. The authors argue pretrained generalist policies could be finetuned with little in-domain data, and that earlier ones fixed the inputs users could supply, lacked effective finetuning or were not public.

Octo is third-party research, and Firsthand is not part of it, endorsed by it or affiliated with its authors. Firsthand makes no claim that its data trained or was used to evaluate Octo, and no performance claim against it. In the sources, the training data is robot trajectories from Open X-Embodiment, and egocentric human video is not described as part of the mix. What Firsthand states is custom egocentric collection for policy pretraining: egocentric video, depth, 3D hand pose and action segments, collected fast, on demand and to a written skill spec. Firsthand does not capture robot action data.

Tokenizers turn the task and observations into tokens: a pretrained t5-base model encodes language, and a shallow convolution stack splits camera and goal images into patches. A transformer backbone with block-wise attention processes them, and learned readout tokens feed a lightweight diffusion action head that predicts a chunk of consecutive actions. The authors say new observations or action heads can be added during finetuning without modifying the pretrained transformer weights.

The authors train on a mixture of 25 datasets from Open X-Embodiment, curating 800k robot episodes from the approximately 1.5M it holds. They removed datasets with no image streams, without delta end-effector control, or that were too repetitive, low resolution or niche.

The paper reports that only 27% of the data has wrist camera information and 56% has language annotations. It describes the source as robot demonstrations, and egocentric human video is not described as part of Octo’s mix.

Octo is not built on a large pretrained language model: its checkpoints are 27M and 93M parameters, with a diffusion-based action head. The project page calls it a transformer-based diffusion policy, so it sits between the OpenVLA and diffusion-policy entries. The table sets out only what each entry’s official sources state, and Firsthand ranks none of them.

In zero-shot tests on setups matching the pretraining data, the paper reports a 29% higher average success rate than RT-1-X, and similar performance to RT-2-X, a 55 billion parameter model. In finetuning with around 100 demonstrations per domain, it reports beating the next best baseline by 52% on average across six setups.

The paper lists limits: heavy degradation on novel behaviors such as flipping or precise insertion, and weak use of wrist camera information. These are the authors’ results on their own robots, and Firsthand draws no comparison from them.

The paper says finetuning used around 100 trajectories for 50k steps, and takes approximately 5 hours on a single 24GB GPU for the base model. The README states the code is under the MIT License, and the paper says checkpoints, scripts and the pretraining pipeline are released. Pretraining dataset licences are Not stated in the sources read.

The paper says robot data is hard to get at scale and often takes significant hardware and human labor, and its ablation reports that performance rose as more datasets joined the mix.

Octo learns from robot actions, which Firsthand does not capture. Firsthand makes no claim that its data finetunes Octo, and mapping its fields to Octo’s action space is Not stated.

Write the target skills, environments and signals into a skill spec and confirm them on a Spec Pilot, a small fast first batch. This is on-demand custom collection, not a catalog download.

Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks. Billing is per validated hour.

What is Octo?

An open-source generalist robot policy for manipulation: a transformer with a diffusion action head, pretrained on 800k robot trajectories curated from Open X-Embodiment.

What data was Octo trained on?

A mixture of 25 Open X-Embodiment datasets, curated to 800k robot trajectories.

How big is Octo?

Two checkpoints are released: Octo-Small with 27M parameters and Octo-Base with 93M parameters.

Is Octo open source?

The code is under the MIT License per the README, and checkpoints are released. Pretraining dataset licences are Not stated in the sources read.

Is Octo a diffusion policy?

The project page describes Octo models as transformer-based diffusion policies. See the diffusion policy entry for the general idea.

Was Octo trained on egocentric human video?

The sources describe robot trajectories from Open X-Embodiment. Egocentric human video is not described as part of its training mix in the sources read.

Is Firsthand data part of Octo?

No. Firsthand is not part of it, endorsed by it or affiliated with its authors, and makes no claim that its data trained or evaluated Octo.