What Is Open X-Embodiment?

Open X-Embodiment is a community effort described in the paper Open X-Embodiment: Robotic Learning Datasets and RT-X Models and on its project page. The paper argues that generalizable robot policies require X-embodiment training: data from multiple robotic platforms. It focuses on robotic manipulation. The repository provides two open-source resources: the dataset and a selection of pre-trained RT-X checkpoints.

The Open X-Embodiment dataset is real robot trajectories, so it carries no egocentric human video, and Firsthand is not part of it, endorsed by it or related to its authors. What the Firsthand site states is the human side of a cross-embodiment mix: egocentric video, depth, 3D hand pose and action segments, collected fast, on demand and to a written skill spec, under a perpetual, buyer-owned commercial license. Firsthand does not capture robot action data.

RT-X is the paper’s name for models trained on its robotics data mixture. It trains two existing architectures, RT-1 and RT-2, and calls the results RT-1-X and RT-2-X. Both take images and a text instruction as input and output an action: a 7-dimensional end-effector vector of x, y, z, roll, pitch, yaw and gripper opening, or the rates of these quantities.

The mixture used in those experiments covers 9 manipulators, fewer than the 22 in the full dataset, because the dataset kept growing after the experiments were run.

The paper and project page state 1M+ real robot trajectories spanning 22 embodiments, from single robot arms to bi-manual robots and quadrupeds, pooled from 60 existing datasets from 34 robotic research labs. The abstract describes a collaboration of 21 institutions demonstrating 527 skills (160,266 tasks).

The paper finds most skills belong to the pick-place family, with a long tail such as wiping or assembling. Hour counts and the licence are Not stated in the sources read for this page, so check the repository before use.

The paper states the data uses the RLDS format, saved as serialized tfrecord files, which accommodates different action spaces and input modalities, such as differing numbers of RGB cameras, depth cameras and point clouds.

Firsthand lists RLDS among its delivery formats, alongside LeRobot, WebDataset and HDF5 / zarr. A shared format does not make datasets interchangeable: the paper notes that camera observations vary substantially across datasets and that the same action vector may induce very different motions on different robots. The site states no compatibility with Open X-Embodiment.

The paper reports positive transfer. On small-dataset domains, RT-1-X outperformed the Original Method on 4 of 5 datasets, with a mean success rate 50% higher than either the Original Method or RT-1. On large-dataset domains RT-1-X did not beat RT-1, which the paper reads as underfitting, while the larger RT-2-X did better. On emergent-skill tests RT-2-X outperformed RT-2 by about 3×.

The paper lists limits: its experiments do not consider robots with very different sensing and actuation modalities, and do not study generalization to new robots. Firsthand makes no performance claim against these results.

Write the target skills, environments and signals into a skill spec and confirm them on a Spec Pilot, a small fast first batch. This is on-demand custom collection, not a catalog download.

Paid, consented contributors perform the tasks. Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks. Billing is per validated hour.

What is Open X-Embodiment?

An open-source repository of 1M+ real robot trajectories from 22 embodiments, plus pre-trained RT-X checkpoints.

What is RT-X?

The paper’s name for models trained on its robotics data mixture: RT-1-X and RT-2-X.

What format is the dataset stored in?

RLDS, saved as serialized tfrecord files, per the paper.

Is Firsthand data part of Open X-Embodiment?

No. Firsthand is not part of it or endorsed by it, and the site states no default compatibility. Firsthand delivers RLDS among its formats.

Does Open X-Embodiment include egocentric human video?

The sources read describe real robot trajectories. They do not describe egocentric human video as part of the dataset.

How fast can custom pretraining data be collected?

Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks.