What Is CrossFormer?

CrossFormer is introduced in Scaling Cross-Embodied Learning: One Policy for Manipulation, Navigation, Locomotion and Aviation (arXiv 2408.11812) by Ria Doshi, Homer Walke, Oier Mees, Sudeep Dasari and Sergey Levine. The project page lists UC Berkeley and Carnegie Mellon University as affiliations and an oral presentation at CoRL 2024. The abstract states the problem: each robotic platform and task might have only a small dataset, and training one policy on multi-robot data is hard because robots have widely varying sensors, actuators and control frequencies.

Firsthand is not affiliated with the CrossFormer authors and makes no claim that any model, CrossFormer included, was trained on Firsthand data. Firsthand captures human egocentric video, not robot data: custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license. Robot trajectories, teleoperation data, navigation, locomotion or aviation data, multi-embodiment datasets, action labels in robot action spaces, success or failure labels, rollouts and language annotations beyond one task instruction per episode are not Firsthand deliverables.

Per the paper, CrossFormer casts cross-embodied imitation learning as a sequence-to-sequence problem. Observations and a task specification are tokenized by modality-specific tokenizers, assembled into one token sequence and fed to a causal, decoder-only transformer backbone shared across all embodiments. The output embeddings go to separate action heads for each class of embodiment, so each robot gets actions of the right dimension.

The practical consequence is that a robot can have any number of camera views or proprioceptive sensors, and the model can predict actions of any dimension. The authors say this removes the need to manually align observation or action spaces, which earlier cross-embodiment work did. The task can be given as a language instruction or a goal image.

The paper describes five observation types: a workspace image, a navigation image, a wrist image, quadruped proprioception and bimanual proprioception. Images are processed with ResNet encoders, and encoder weights are shared for camera views of the same type. Language instructions are processed jointly with image observations, and goal images are stacked on the current image.

Special readout tokens follow the observations at each time-step, and action heads on those tokens predict continuous actions with an L1 loss. The policy has four action heads and can predict chunks of actions, which the authors say matters for embodiments with high control frequencies. Per the paper, the same design covers a bimanual ALOHA system and ground and aerial navigation robots with 2D waypoint control.

The authors train on 900K trajectories. Per the paper, that is a subset of the Open X-Embodiment dataset, which aggregates 1.5 million episodes, plus the GNM navigation and DROID manipulation datasets and Go1 quadruped and ALOHA bimanual data collected for the project. The Go1 data came from rolling out an expert policy trained with reinforcement learning in simulation.

The paper counts 20 robot embodiments in the training mixture, while the project page abstract says 30. This page treats the embodiment count as the authors’ own figure and does not settle the difference. All of this is robot-collected data, which is a different kind of data from egocentric human video.

The authors evaluate on single-arm manipulation with a WidowX and a Franka, bimanual manipulation with ALOHA, navigation with a LoCoBot, locomotion with a Unitree Go1 and flight with a Tello quadcopter. They report that CrossFormer matches the performance of the same architecture trained on only the target robot and of the best prior method in each setting, and outperforms a prior cross-embodiment method that aligns action spaces.

The paper states that no quadcopter data was used in training, so the Tello result is zero-shot. This page repeats none of the authors’ success rates or scores; read them in the paper.

The authors list three. Their results do not yet show significant positive transfer across embodiments. The data mix uses hand-picked sampling weights. And inference speed can become a limit for high-frequency robots as the model grows. Future work in the paper includes exploring more diverse sources such as sub-optimal robot data or action-free human video.

The sources read do not state whether code or weights are released, so this page links none.

The comparison table below sets CrossFormer beside four site entries. The paper discusses Octo and Open X-Embodiment directly, and cites Gato and RoboCat as prior flexible transformer policies. It does not mention HPT, which is a separate paper, so that row says Not stated.

In short, CrossFormer is one policy architecture and trained model, Open X-Embodiment is a dataset it draws on, Octo is a prior generalist policy, and cross-embodiment learning is the general training idea. This page ranks no models.

Per the paper, CrossFormer learns from robot trajectories: observations paired with actions in each embodiment’s own action space, with a language instruction or goal image per task. Hindsight goal relabeling supplies goal images from the trajectories themselves.

Firsthand states human egocentric capture with stereo-derived depth, 21-joint hand pose and one task instruction per episode, and no robot actions. Whether those streams suit CrossFormer is Not stated, and no Firsthand data is claimed to have trained or been used by CrossFormer.

When a program needs real human egocentric demonstrations that public data does not cover. Firsthand does not build or train models. Confirm the fields on a Spec Pilot, a small fast first batch; the median from signed spec to first validated batch is 48 hours where coverage is deep, three to four weeks where thin.

What is CrossFormer?

A published transformer policy (arXiv 2408.11812) that controls single and dual arm manipulators, wheeled robots, quadcopters and quadrupeds with the same network weights, without manually aligning observation or action spaces. It is third-party research.

What is a cross-embodied policy?

One policy trained on data from many different robots, despite their different sensors, actuators and control frequencies. CrossFormer handles this by tokenizing observations into one sequence and using separate action heads per class of embodiment.

What data was CrossFormer trained on?

Per the authors, 900K robot trajectories drawn from Open X-Embodiment, GNM and DROID plus Go1 and ALOHA data collected for the project. The paper counts 20 embodiments and the project page says 30. It is robot data, not egocentric human video.

Are the CrossFormer code and weights available?

The sources read do not state a code or weights release, so this page links none. Check the official project page for the authors’ current materials.

Does Firsthand use CrossFormer, and can I request a custom dataset?

Firsthand is not affiliated with the CrossFormer authors and claims no use, training or compatibility. Robot trajectories, action labels, navigation or locomotion data, rollouts and success or failure labels are not Firsthand deliverables. You can request a custom dataset.