What Is RDT-1B?

RDT-1B is introduced in RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation (arXiv 2410.07864), with an official project page, GitHub repository and Hugging Face model card. The paper presents the Robotics Diffusion Transformer (RDT) and names two reasons bimanual foundation models are hard: coordinating two arms produces multi-modal action distributions, and training data for dual-arm robots is scarce. A diffusion model can represent several valid ways of doing a task instead of averaging them into one infeasible action, which is why the authors chose it. The paper abstract says the model was scaled up to 1.2B parameters, while the README and model card describe it as a 1B-parameter model, so this page uses the name RDT-1B and does not pick between the two figures. The authors state that their goal is to use multi-robot data to improve generalisation in bimanual manipulation, not to build a cross-embodiment model for every robot.

RDT-1B is third-party research. Firsthand is not affiliated with its authors or their institution, and makes no claim that RDT-1B or any other named model was trained, fine-tuned or evaluated on Firsthand data. The site states custom egocentric collection to a written spec, fast and on demand, delivering first-person human video with 3D hand pose at 21 joints per hand and a per-joint visibility flag, calibrated stereo depth and 6-DoF IMUs at 200 Hz. The only language data tied to Firsthand capture is one task instruction per episode. Firsthand captures first-person human video, not robot or teleoperation data.

Per the paper, RDT models the distribution of an action given a language instruction and an observation, rather than regressing a single action. At run time it starts from random noise and removes it over several denoising steps to produce an action chunk, a sequence of future actions, which the authors say encourages temporal consistency and reduces error build-up. The README and model card state that, given a language instruction and RGB images from up to three views, RDT predicts the next 64 robot actions. The paper’s framework figure lists the conditioning images as an exterior camera plus right and left wrist cameras, and the inputs also include the robot’s proprioception and the control frequency.

Each input is encoded into a shared latent space. The paper uses SigLIP as an image-text-aligned vision encoder and T5-XXL as the language model, both with frozen weights, and MLPs with Fourier features for low-dimensional signals. The backbone is a Diffusion Transformer with three stated changes: QKNorm and RMSNorm for numerical stability, a nonlinear MLP decoder, and alternating condition injection, which alternates image and text tokens across successive cross-attention layers so that image tokens do not overshadow the instruction. For real-time use the paper adopts DPM-Solver++, which it says cuts the diffusion steps from 100 to 5.

Because dual-arm robot data is scarce, the paper pre-trains on multi-robot data first. The collection has 46 datasets of various robots with more than 1M trajectories and 21 TB in total. The Hugging Face model card lists the RT-1 Dataset, RH20T, DROID, BridgeData V2, RoboSet and a subset of Open X-Embodiment among them. The paper describes this data as mostly single-arm. To use it together, the authors define a Physically Interpretable Unified Action Space, in which each element of a robot’s action vector is placed in the position that matches its physical meaning and the remaining positions are padded.

The paper then fine-tunes on a self-collected bimanual dataset recorded on the ALOHA dual-arm robot. It reports 6K+ trajectories covering 300+ tasks, 100+ objects and 15+ rooms with different lighting, and says GPT-4-Turbo was used to rewrite human-annotated instructions to increase text diversity. The paper compares RDT against ACT, OpenVLA and Octo in real-robot experiments, and states that it generalises zero-shot to unseen objects and scenes, follows language instructions and learns new skills from 1 to 5 demonstrations.

It depends on the definition. The Hugging Face model card lists the task type as Vision-Language-Action (language, image to robot actions) and the model type as Diffusion Policy with Transformers. The paper instead describes RDT as a language-conditioned visuomotor policy and contrasts it with approaches that adapt large vision-language models to predict actions directly, which it says run into quantization errors because they discretise the action space.

RDT keeps actions continuous and uses its pretrained encoders only to condition the denoising network. So it takes language and images in and produces robot actions out, and its model card labels it a VLA, but it does not generate actions as language-model tokens. The VLA versus diffusion policy comparison sets the two approaches side by side.

The repository describes fine-tuning on your own dataset in three parts: register the dataset, reimplement the dataset loader class so it parses your files, and fill your robot’s action into the unified action vector. A single-arm robot’s action goes into the right-arm portion, end-effector rotation uses a 6D representation, and the README advises against normalising physical quantities other than gripper width. It recommends training for at least 150K steps, and a smaller RDT-170M checkpoint when VRAM is limited.

The model card says RDT cannot yet generalise to new robot platforms that were not seen in pre-training because of the embodiment gap, and recommends collecting a small dataset on the target robot to fine-tune on. The README and the model card both state that the code, pre-trained weights and data are licensed under MIT. Read the licence text itself before a commercial program relies on it.

RDT-1B is trained on robot data: images, robot proprioception, robot actions and a language instruction per trajectory. Firsthand captures first-person human video, not robot or teleoperation data, so it does not deliver the robot action trajectories or proprioception that RDT’s fine-tuning loader expects. Whether human egocentric data would improve a policy like RDT is Not stated in the sources read, and this page draws no conclusion about it.

What the site does state is custom egocentric collection, fast and on demand, to a written spec. For two-handed work, both hands stay in frame with per-hand 3D pose and visibility flags. The only language data is one task instruction per episode. Confirm the spec on a Spec Pilot, a small fast first batch. Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks. Billing is per validated hour, and delivery is in the format your training stack already reads: RLDS, LeRobot, WebDataset or HDF5.

What is RDT-1B?

The Robotics Diffusion Transformer, a diffusion foundation model for bimanual manipulation from Tsinghua University. It takes a language instruction, up to three RGB views and the robot state, and predicts the next 64 robot actions by denoising.

Is RDT-1B a VLA model?

Its Hugging Face model card lists the task type as Vision-Language-Action, and the model type as Diffusion Policy with Transformers. The paper calls it a language-conditioned visuomotor policy that keeps actions continuous instead of discretising them.

How many parameters does RDT-1B have?

The paper abstract says it was scaled up to 1.2B parameters, while the README and model card describe a 1B-parameter model. The README also lists a smaller RDT-170M checkpoint.

What data was RDT-1B trained on?

Per the paper, pre-training used 46 multi-robot datasets with 1M+ trajectories and 21 TB, and fine-tuning used 6K+ self-collected bimanual trajectories on the ALOHA dual-arm robot.

Is RDT-1B open source, and what is the licence?

The README and the Hugging Face model card state that the code, pre-trained weights and data are licensed under MIT. Read the licence text before a commercial program relies on it.

Did RDT-1B use Firsthand data, and can I request a custom dataset?

No. Firsthand is not affiliated with the RDT-1B authors and makes no claim that RDT-1B or any model was trained on Firsthand data. You can request a custom egocentric dataset: write the tasks, fields and format into a spec, confirm it on a Spec Pilot, and billing is per validated hour. Firsthand captures first-person human video, not robot or teleoperation data.