What Is RT-1 (Robotics Transformer)?

RT-1 is introduced in RT-1: Robotics Transformer for Real-World Control at Scale (arXiv 2212.06817), submitted 13 December 2022. The authors argue that general robotic models need open-ended, task-agnostic training and high-capacity architectures that can absorb diverse robot data, and they study generalization against data size, model size and data diversity on real robots.

RT-1 is third-party research. Firsthand is not part of it, endorsed by it or affiliated with its authors, and makes no claim that RT-1 or any other model uses or was trained on Firsthand data. The sources read describe robot demonstrations, not egocentric human video. Firsthand states custom egocentric collection to a written spec: video, a paired instruction and the action that carried it out, on one hardware-triggered clock.

RT-1 takes a short sequence of images and a natural-language task description and outputs an action at each time step. An ImageNet-pretrained EfficientNet, conditioned on an instruction embedding through FiLM layers, extracts task-relevant features. A TokenLearner compresses them to a compact set of tokens, and a Transformer produces discretized action tokens.

The action has seven arm dimensions, three base dimensions and one discrete mode that switches between arm, base or ending the episode. Control is closed-loop at 3 Hz until a terminate action or a pre-set step limit.

The authors collected over 130k real-world episodes covering over 700 tasks, with a fleet of 13 robots over 17 months. The listed skills include picking, placing, opening and closing drawers, getting items in and out of drawers, and opening jars. Most of the data came from a mock kitchen, which the RT-2 entry on this site describes as an office kitchen environment.

The RT-2 entry on this site states that RT-2 builds on the protocol of RT-1, uses RT-1 robot demonstrations as its robot data, and reports more than a 3x average success rate over RT-1 on emergent-skill tests.

A vision-language-action (VLA) model, as defined on this site, maps a visual scene and a language instruction to an action. The sources read do not call RT-1 a VLA model, so this page does not either.

The authors note that real-world robot data is hard to collect. RT-1’s came from robots running for 17 months, each demonstration tied to a language instruction. The VLA entry on this site says such a mapping needs frames, instruction and action on a timeline where the three are provably simultaneous.

Firsthand does not claim that egocentric human video replaces robot demonstrations or trains RT-1. It collects egocentric video with paired instruction and action signals on demand, to a written spec, for skills or environments not in an open dataset.

Confirm the spec on a Spec Pilot. The median from signed spec to first validated batch is 48 hours where coverage is deep and three to four weeks where it is thin. Billing is per validated hour.

What is RT-1?

A Transformer-based robot control model that maps images and a task description to a robot action at each time step.

How does RT-1 work?

EfficientNet with FiLM conditioning extracts features, TokenLearner compresses them, and a Transformer outputs discretized action tokens at 3 Hz.

What data was RT-1 trained on?

Over 130k real-world robot episodes covering over 700 tasks, from 13 robots over 17 months.

Is RT-1 open source?

The project page says so, and the official repository holds code and checkpoints under the Apache license. The terms of the data are Not stated in the sources read.

Was RT-1 trained on egocentric human video?

Not described in the sources read. The data is robot demonstrations.

Is Firsthand data part of RT-1?

No. Firsthand is not part of it or affiliated with its authors, and makes no claim that any model was trained on Firsthand data.