Now recruiting capture teams worldwide. Get paid to record first-person video of real work.

Apply to capture

Glossary

What Is H-RDT (Human Manipulation Enhanced Bimanual Robotic Manipulation)?

H-RDT pretrains a bimanual robot policy on egocentric human video with 3D hand pose, then fine-tunes on robot data. Firsthand is not affiliated.

Last reviewed: November 2026

Short answer

H-RDT, short for Human to Robotics Diffusion Transformer, is the method from the paper “H-RDT: Human Manipulation Enhanced Bimanual Robotic Manipulation” (arXiv 2507.23523). It pretrains a 2B-parameter diffusion transformer on egocentric human manipulation video with paired 3D hand pose, then fine-tunes it on robot data for bimanual manipulation. Third-party research; Firsthand is not affiliated.

01

What is H-RDT, and what problem does it address?

H-RDT is introduced in H-RDT: Human Manipulation Enhanced Bimanual Robotic Manipulation (arXiv 2507.23523) by Hongzhe Bi, Lingxuan Wu, Tianwei Lin, Hengkai Tan, Zhizhong Su, Hang Su and Jun Zhu; the project page lists Tsinghua University and Horizon Robotics. The authors argue that imitation learning is limited by scarce large-scale, high-quality robot demonstration data, and that pretraining on cross-embodiment robot datasets is hard because robots differ in morphology and action space. Their alternative is human manipulation pretraining for bimanual robots: egocentric human video with paired 3D hand pose as a behavioral prior, followed by a robot-specific stage. The paper says it builds on the RDT architecture covered in the RDT-1B entry.

02

Where does Firsthand fit next to H-RDT?

H-RDT is third-party research. Firsthand is not affiliated with its authors, and no named model, H-RDT included, was trained on Firsthand data. The site states custom egocentric collection to a written spec, fast and on demand, delivering first-person human video with 3D hand pose at 21 joints per hand and a per-joint visibility flag, calibrated stereo depth and 6-DoF IMUs at 200 Hz. The only language data tied to Firsthand capture is one task instruction per episode. Firsthand captures first-person human video, not robot or teleoperation data, so H-RDT’s robot fine-tuning stage is outside what it delivers.

03

How does H-RDT go from egocentric human video to a bimanual policy?

Per the paper, training has two stages. Stage 1 pretrains on the complete EgoDex dataset, which the authors describe as 338K+ trajectories across 194 manipulation tasks, using a 48-dimensional human hand action. Stage 2 is cross-embodiment fine-tuning: the vision encoder, language encoder and transformer backbone weights carry over, while the state adaptor, action adaptor and action decoder are re-initialised for the target robot’s action space.

The model is a diffusion transformer with 2B parameters trained with flow matching. The authors use DinoV2 and SigLIP for images and a T5-XXL language model, inject image and language features through separate cross-attention, and state that the design builds on RDT.

04

What is the 48-dimensional human action representation?

The authors represent each human action as 48 numbers: bilateral wrist poses, with 3D position and 6D orientation for both hands (18 dimensions), plus 3D fingertip positions for all fingers on both hands (30 dimensions). They say the wrist poses match robot end-effector poses, so the vector is a superset of the action space of most end-effector-controlled robots.

Firsthand states no such vector. The site states 3D hand pose at 21 joints per hand with a per-joint visibility flag, and the second table below sets the two side by side.

05

What did the authors report, and where does H-RDT struggle?

The abstract reports that H-RDT outperforms training from scratch and methods including π0 and RDT, with improvements of 13.9% in simulation and 40.5% in real-world experiments over training from scratch. In a few-shot test on a dual-arm ARX5 robot, 113 pick-and-place tasks with 1 to 5 demonstrations each, the authors report an average success of 41.6% for H-RDT against 16.0% for RDT, 31.2% for π0 and 17.6% for H-RDT without human pretraining.

The same paper reports single-task RoboTwin 2.0 averages of 68.7% in Easy mode and 25.6% in Hard mode, which adds domain randomization, so Hard mode stays far lower. These are the authors’ figures on their own setups; Firsthand has not verified them and quotes them only for context.

06

What hardware does the H-RDT paper name?

The paper names robots rather than capture rigs for its human data. Simulation covers Aloha-Agilex-1.0 and Franka-Panda; real-world validation covers a dual-arm ARX5, Aloha-Agilex-2.0 (dual-arm Piper) and a dual UR5 with demonstrations collected through the Universal Manipulation Interface, which the authors describe as a framework for portable human demonstration collection with hand-held grippers.

The human pretraining data comes from EgoDex, whose capture the EgoDex entry describes as one consumer headset with bare hands. Firsthand does not claim to use any of this hardware.

07

How does H-RDT compare with EgoMimic, HAT, RDT-1B and cross-embodiment learning?

The authors contrast H-RDT with EgoMimic, which they say trains on 2k human demonstrations and needs paired robot data during co-training, and with HAT, which they say trains on 27k demonstrations and assumes humanoid kinematics. They say H-RDT instead decouples large-scale human pretraining from robot-specific fine-tuning and adapts to other robots through modular action adapters. Those are the authors’ characterisations; the first table maps each to its site entry.

08

What data does an H-RDT-style approach need, and what does Firsthand state?

The pretraining stage consumes egocentric human video with 3D hand pose, and the fine-tuning stage consumes robot data for the target embodiment. Firsthand’s stated scope covers only the human side, as custom collection: first-person video, depth, 3D hand pose and one task instruction per episode, confirmed first on a Spec Pilot. Robot trajectories, retargeted actions and action labels are not Firsthand deliverables, and the site makes no claim that its data suits any H-RDT pipeline.

Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks. Billing is per validated hour.

09

H-RDT at a glance, as stated in the paper

H-RDT at a glance, as stated in the paper
ItemWhat the paper states
PaperH-RDT: Human Manipulation Enhanced Bimanual Robotic Manipulation (arXiv 2507.23523)
Authors and affiliationsHongzhe Bi, Lingxuan Wu, Tianwei Lin, Hengkai Tan, Zhizhong Su, Hang Su, Jun Zhu; Tsinghua University and Horizon Robotics (project page)
Method nameH-RDT, Human to Robotics Diffusion Transformer
ModelDiffusion transformer, 2B parameters, flow matching
Stage 1Pretraining on the complete EgoDex dataset, 338K+ trajectories, 194 tasks, 48-dimensional human hand action
Stage 2Cross-embodiment fine-tuning with re-initialised state adaptor, action adaptor and action decoder
Robots namedSimulation: Aloha-Agilex-1.0 and Franka-Panda. Real world: dual-arm ARX5, Aloha-Agilex-2.0 (dual-arm Piper) and UR5 with UMI
Simulation benchmarkRoboTwin 2.0
Code and weightsOfficial GitHub repository (MPL-2.0 per GitHub’s licence field) and Hugging Face model page; weights licence Not stated in the sources read
Firsthand relationshipNone; no named model was trained on Firsthand data

11

H-RDT beside the RDT-1B, EgoDex, EgoMimic, HAT and cross-embodiment entries

H-RDT beside the RDT-1B, EgoDex, EgoMimic, HAT and cross-embodiment entries
Site entryWhat the entry statesHow H-RDT differs (sources read)
RDT-1BOpen diffusion transformer policy for bimanual manipulation, pre-trained on multi-robot data then fine-tuned on a target dual-arm robotH-RDT says it builds on the RDT architecture but pretrains on human egocentric video with 3D hand pose before robot fine-tuning
EgoDex datasetPublic egocentric dexterous manipulation dataset: 829 hours, 338,000 episodes, 194 tabletop tasks, 25 tracked joints per handH-RDT uses the complete EgoDex dataset for stage 1; EgoDex is the authors’ pretraining source, not a Firsthand dataset
EgoMimicCo-trains one policy on human embodiment data, egocentric video with 3D hand tracking from Project Aria glasses, and robot dataThe H-RDT authors say EgoMimic needs paired robot data during co-training; H-RDT separates human pretraining from robot fine-tuning
HATPairs the PH2D egocentric human-humanoid dataset with a policy whose state-action space is unified across humans and humanoidsThe H-RDT authors say HAT’s retargeting assumes humanoid kinematics; H-RDT targets other morphologies through modular action adapters
Cross-embodiment learningTraining one policy on data from multiple embodiments, robots and human demonstrationsH-RDT pretrains on one human embodiment, then adapts to each robot; the authors contrast this with pretraining on mixed robot datasets

12

What H-RDT consumes beside what the Firsthand site states

What H-RDT consumes beside what the Firsthand site states
AspectH-RDT (sources read)What the Firsthand site states
Pretraining inputEgocentric human video from EgoDex with paired 3D hand pose; one egocentric camera viewCustom first-person human video, collected to a written spec
Hand signal48-dimensional vector: wrist pose (3D position, 6D orientation) and fingertip positions, both hands3D hand pose at 21 joints per hand with a per-joint visibility flag; no 48-dimensional vector stated
Depth and inertialNot listed as pretraining inputsCalibrated stereo depth and 6-DoF IMUs at 200 Hz
LanguageA language instruction conditions the policyOne task instruction per episode
Robot dataRequired for stage 2 on the target robotNot captured; Firsthand captures first-person human video, not robot or teleoperation data
Training useThe authors’ own training and evaluationNo named model was trained on Firsthand data; no affiliation
DeliveryPublic EgoDex releaseSpec Pilot, then billed per validated hour; custom collection, fast and on demand

13

Frequently asked questions

What is H-RDT?
H-RDT, Human to Robotics Diffusion Transformer, is a published method (arXiv 2507.23523) that pretrains a 2B-parameter diffusion transformer on egocentric human manipulation video with 3D hand pose, then fine-tunes it on robot data for bimanual manipulation. It is third-party research.
What data does H-RDT pretrain on?
Per the paper, the complete EgoDex dataset, described as 338K+ trajectories across 194 manipulation tasks, using a 48-dimensional human hand action made of wrist poses and fingertip positions.
Does H-RDT need robot data?
Yes, for stage 2. The paper fine-tunes on robot-specific data with re-initialised action encoders and decoders, so human pretraining reduces but does not remove the need for target-robot data.
What did the authors report for H-RDT?
They report improvements of 13.9% in simulation and 40.5% in real-world experiments over training from scratch, and a 41.6% average on a few-shot dual-arm ARX5 test against 16.0% for RDT and 31.2% for π0. These are the authors’ figures; Firsthand has not verified them.
Is H-RDT open source?
The authors publish an official GitHub repository, which GitHub lists under MPL-2.0, and a Hugging Face model page. The weights licence is Not stated in the sources read, so read the licence text before relying on it commercially.
Was H-RDT trained on Firsthand data?
No. Firsthand is not affiliated with the authors and claims no use, training or compatibility. Robot trajectories, retargeted actions, action labels and a 48-dimensional hand vector are not Firsthand deliverables.
How can I request a custom egocentric dataset for human-to-robot learning?
Request a custom dataset with your objects, interactions and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep, confirmed first on a Spec Pilot.