What Is RT-H?

RT-H is introduced in RT-H: Action Hierarchies Using Language (arXiv 2403.01823) by Suneel Belkhale, Tianli Ding, Ted Xiao, Pierre Sermanet, Quan Vuong, Jonathan Tompson, Yevgen Chebotar, Debidatta Dwibedi and Dorsa Sadigh. The project page lists Google DeepMind Robotics and Stanford University as affiliations. The paper was submitted on 4 March 2024. RT-H builds an action hierarchy out of language: the policy goes from the high-level task to a language motion, and from the language motion to the action, using visual context at every stage.

RT-H is third-party research. Firsthand is not part of it or affiliated with its authors, and makes no claim that any model was trained on Firsthand data. What the site states is custom egocentric collection of real human demonstrations, fast and on demand, to a written spec: 3D hand pose at 21 joints per hand, calibrated stereo depth and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset with a buyer-owned commercial license. The only language data the site states is one task instruction per episode. The site does not state language-motion labels, so this page makes no claim that Firsthand provides them.

Per the abstract, language-conditioned policies share data between semantically similar tasks. As tasks become more semantically diverse, such as “pick coke can” and “pour cup”, sharing data gets harder, and mapping high-level tasks to actions needs much more demonstration data.

The authors’ insight is to teach the robot the language of actions, describing low-level motions with fine-grained phrases like “move arm forward” or “close gripper”. Predicting these language motions between task and action forces the policy to learn the low-level structure shared across disparate tasks.

The project page splits the policy into two queries. The language motion query predicts a fine-grained language motion from the image tokens and the task description tokens. The action query then decodes that language motion into actions, using the context of the task and the scene.

The page says RT-H uses a vision-language model co-trained on internet-scale data to predict both the language motion from the task and the action from the language motion. The same phrase, such as “move arm forward”, plays out differently depending on the task and scene.

Because the action query is conditioned on a language motion, a person can step in with a different one, for example “move arm left” instead of “move arm forward”, and the policy follows it. To learn from corrections, only the language motion query is updated with the newly labelled corrections, and the updated model is deployed back into the hierarchy.

The authors report that policies trained this way can learn from such interventions and outperform methods that learn from teleoperated interventions.

On their Diverse+Kitchen multi-task dataset with eight evaluation tasks, the project page says RT-H outperforms RT-2 by 15% on average, and scores higher on six of the eight tasks. Replacing language with class labels drops performance significantly, and swapping the automated motion labelling for K-Means action clusters causes a minor drop.

For corrections, the page reports a rise from 40% to 63% with 30 intervention episodes per task. These are the authors’ claims on their own setups, and Firsthand does not reproduce them.

The site’s RT-1 entry covers a Transformer that outputs an action from images and a task description, and its RT-2 entry covers a model that writes robot actions as text tokens. RT-H adds an intermediate language-motion step between task and action, and the authors compare it against RT-2 as their baseline. Whether the paper compares against RT-1 is Not stated in the text read.

The site’s Hi Robot entry covers a hierarchical system in which a high-level model outputs a simple language command for a low-level policy. The site’s VLA-OS entry lists RT-H and Hi Robot as examples of the Hierarchical-VLA category, which is VLA-OS’s grouping. How the RT-H authors relate their work to Hi Robot is Not stated in the sources read.

The sources read describe multi-task robot datasets with an automated motion labelling procedure that produces the language motions. Dataset size, robot hardware and the number of demonstrations are Not stated in the sources read.

The sources read do not describe human egocentric video, stereo depth, hand pose or IMU data as part of RT-H’s training. On the site, the only language data is one task instruction per episode.

The authors note that RT-H cannot zero-shot perform the novel tasks they tried, such as unstacking cups, and needs human intervention after the early phases; they present this as a sign of learning shared phases of tasks. Code, weights and licence are Not stated in the sources read.

When a program needs real human egocentric demonstrations of tasks, environments or objects that public data does not cover, specified in writing. Firsthand does not build or train models.

Confirm the spec on a Spec Pilot, a small fast first batch. Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks. Billing is per validated hour.

What is RT-H?

A published robot policy (arXiv 2403.01823) that predicts a fine-grained language motion from the image and task, then predicts the action conditioned on it. It is third-party research.

What are action hierarchies using language?

Per the paper, a two-step policy: task to language motion, such as “move arm forward”, then language motion to action, with visual context at every stage.

Is RT-H better than RT-2?

The authors report RT-H outperforming RT-2 by 15% on average on their eight-task Diverse+Kitchen evaluation. These are the authors’ results on their own setups.

Is RT-H a vision-language-action model?

It maps a visual scene and a language instruction to an action, which fits the site’s definition. The sources read do not use the label themselves.

Does Firsthand data train RT-H?

No claim is made. Firsthand is not affiliated with its authors, and the site does not state that any model was trained on Firsthand data.

Does Firsthand deliver language-motion labels?

The only language data the site states is one task instruction per episode. Language-motion labels are not stated, and scope for any model is confirmed on a Spec Pilot.