What Is RT-2 (Robotics Transformer 2)?

RT-2 is introduced in the paper RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control (arXiv 2307.15818). The authors want a single end-to-end model that maps robot observations to actions while keeping the benefits of large-scale language and vision-language pretraining. They call this category vision-language-action (VLA) models and present RT-2 as an example, building on the protocol of an earlier model, RT-1.

RT-2 is a third-party research model, and Firsthand is not part of it, endorsed by it or affiliated with its authors. Firsthand makes no claim that its data trained or was used to evaluate RT-2, and no performance claim against it. In the paper, the robot data is demonstrations collected with robots, and egocentric human video is not described as part of the training mix. What Firsthand states is custom egocentric collection for VLA-style training: synchronized egocentric video, a paired task instruction and the action that carried it out (3D hand pose, gripper state or a robot action log), hardware-triggered onto one clock, to a written spec, on demand.

The paper’s action space is a 6-DoF positional and rotational displacement of the robot end-effector, the level of gripper extension, and a discrete command that ends the episode. The continuous dimensions are discretized into 256 bins, so one action becomes 8 integers, and 256 tokens in the model are reserved to stand for the bins. An action is then written as a string of numbers, such as “1 128 91 241 5 101 127”.

The model input is a robot camera image plus a text task description. At inference the output tokens are de-tokenized back into robot actions for closed-loop control.

Training combines two sources. The web data is visual question answering, captioning and interleaved image-text examples. The robot data is demonstration episodes from the RT-1 paper, which the authors state was collected with 13 robots over 17 months in an office kitchen environment.

Each demonstration is annotated with a natural-language instruction made of a verb for the skill and one or more nouns for the objects. The recipe is co-fine-tuning, meaning robot data is trained together with the original web data. The appendix states that robot data makes up about 50% of the mixture for one variant and about 66% for the other.

The paper trains two instantiations: RT-2-PaLI-X, built from 5B and 55B parameter models, and RT-2-PaLM-E, built from a 12B parameter model.

The authors deploy these models in a multi-TPU cloud service queried over the network, and state that the 55B model runs at 1 to 3 Hz and the 5B model at around 5 Hz.

The authors ran about 6,000 evaluation trials. They report roughly a 2x improvement in generalization to new objects, backgrounds and environments over the next two baselines, RT-1 and MOO, and more than a 3x average success rate over RT-1 on emergent-skill tests covering symbol understanding, reasoning and human recognition.

They also report that co-fine-tuning generalized better than fine-tuning on robot data alone. These are the authors’ results on their own setup, and Firsthand draws no comparison from them.

The authors state that the robot does not acquire new motions from the web data: its physical skills stay limited to the distribution of skills in the robot data, and it deploys them in new ways. Their listed failure cases include grasping objects by a specific part such as a handle, novel motions such as wiping with a towel or tool use, and dexterous motions such as folding a towel.

The limitations section names new data collection paradigms, such as videos of humans, as a direction for future work. The paper does not evaluate that, and this page makes no claim about it.

Write the target skills, environments and signals into a skill spec and confirm them on a Spec Pilot, a small fast first batch. This is on-demand custom collection, not a catalog download.

Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks. Billing is per validated hour.

What is RT-2?

A vision-language-action model from Google DeepMind authors that co-fine-tunes a pretrained vision-language model on robot trajectories and web data, writing robot actions as text tokens.

How does RT-2 represent robot actions?

As text. Each action is discretized into 256 bins across its dimensions, giving 8 integers that are written as a string of tokens and de-tokenized back into robot actions at inference.

What data was RT-2 trained on?

Web visual question answering, captioning and image-text data, co-fine-tuned with RT-1 robot demonstrations collected with 13 robots over 17 months in an office kitchen environment.

Was RT-2 trained on egocentric human video?

The paper describes web vision-language data and robot demonstrations. Egocentric human video is not described as part of its training mix in the sources read.

Are RT-2 weights or code available?

Not stated in the sources read. Check the official project page for any later update.

Is Firsthand data part of RT-2?

No. Firsthand is not part of it, endorsed by it or affiliated with its authors, and makes no claim that its data trained or evaluated RT-2.