Glossary
What Is H-RDT (Human Manipulation Enhanced Bimanual Robotic Manipulation)?
Last reviewed: November 2026
Short answer
H-RDT, short for Human to Robotics Diffusion Transformer, is the method from the paper “H-RDT: Human Manipulation Enhanced Bimanual Robotic Manipulation” (arXiv 2507.23523). It pretrains a 2B-parameter diffusion transformer on egocentric human manipulation video with paired 3D hand pose, then fine-tunes it on robot data for bimanual manipulation. Third-party research; Firsthand is not affiliated.
01
What is H-RDT, and what problem does it address?
H-RDT is introduced in H-RDT: Human Manipulation Enhanced Bimanual Robotic Manipulation (arXiv 2507.23523) by Hongzhe Bi, Lingxuan Wu, Tianwei Lin, Hengkai Tan, Zhizhong Su, Hang Su and Jun Zhu; the project page lists Tsinghua University and Horizon Robotics. The authors argue that imitation learning is limited by scarce large-scale, high-quality robot demonstration data, and that pretraining on cross-embodiment robot datasets is hard because robots differ in morphology and action space. Their alternative is human manipulation pretraining for bimanual robots: egocentric human video with paired 3D hand pose as a behavioral prior, followed by a robot-specific stage. The paper says it builds on the RDT architecture covered in the RDT-1B entry.
02
Where does Firsthand fit next to H-RDT?
H-RDT is third-party research. Firsthand is not affiliated with its authors, and no named model, H-RDT included, was trained on Firsthand data. The site states custom egocentric collection to a written spec, fast and on demand, delivering first-person human video with 3D hand pose at 21 joints per hand and a per-joint visibility flag, calibrated stereo depth and 6-DoF IMUs at 200 Hz. The only language data tied to Firsthand capture is one task instruction per episode. Firsthand captures first-person human video, not robot or teleoperation data, so H-RDT’s robot fine-tuning stage is outside what it delivers.
03
How does H-RDT go from egocentric human video to a bimanual policy?
Per the paper, training has two stages. Stage 1 pretrains on the complete EgoDex dataset, which the authors describe as 338K+ trajectories across 194 manipulation tasks, using a 48-dimensional human hand action. Stage 2 is cross-embodiment fine-tuning: the vision encoder, language encoder and transformer backbone weights carry over, while the state adaptor, action adaptor and action decoder are re-initialised for the target robot’s action space.
The model is a diffusion transformer with 2B parameters trained with flow matching. The authors use DinoV2 and SigLIP for images and a T5-XXL language model, inject image and language features through separate cross-attention, and state that the design builds on RDT.
04
What is the 48-dimensional human action representation?
The authors represent each human action as 48 numbers: bilateral wrist poses, with 3D position and 6D orientation for both hands (18 dimensions), plus 3D fingertip positions for all fingers on both hands (30 dimensions). They say the wrist poses match robot end-effector poses, so the vector is a superset of the action space of most end-effector-controlled robots.
Firsthand states no such vector. The site states 3D hand pose at 21 joints per hand with a per-joint visibility flag, and the second table below sets the two side by side.
05
What did the authors report, and where does H-RDT struggle?
The abstract reports that H-RDT outperforms training from scratch and methods including π0 and RDT, with improvements of 13.9% in simulation and 40.5% in real-world experiments over training from scratch. In a few-shot test on a dual-arm ARX5 robot, 113 pick-and-place tasks with 1 to 5 demonstrations each, the authors report an average success of 41.6% for H-RDT against 16.0% for RDT, 31.2% for π0 and 17.6% for H-RDT without human pretraining.
The same paper reports single-task RoboTwin 2.0 averages of 68.7% in Easy mode and 25.6% in Hard mode, which adds domain randomization, so Hard mode stays far lower. These are the authors’ figures on their own setups; Firsthand has not verified them and quotes them only for context.
06
What hardware does the H-RDT paper name?
The paper names robots rather than capture rigs for its human data. Simulation covers Aloha-Agilex-1.0 and Franka-Panda; real-world validation covers a dual-arm ARX5, Aloha-Agilex-2.0 (dual-arm Piper) and a dual UR5 with demonstrations collected through the Universal Manipulation Interface, which the authors describe as a framework for portable human demonstration collection with hand-held grippers.
The human pretraining data comes from EgoDex, whose capture the EgoDex entry describes as one consumer headset with bare hands. Firsthand does not claim to use any of this hardware.
07
How does H-RDT compare with EgoMimic, HAT, RDT-1B and cross-embodiment learning?
The authors contrast H-RDT with EgoMimic, which they say trains on 2k human demonstrations and needs paired robot data during co-training, and with HAT, which they say trains on 27k demonstrations and assumes humanoid kinematics. They say H-RDT instead decouples large-scale human pretraining from robot-specific fine-tuning and adapts to other robots through modular action adapters. Those are the authors’ characterisations; the first table maps each to its site entry.
08
What data does an H-RDT-style approach need, and what does Firsthand state?
The pretraining stage consumes egocentric human video with 3D hand pose, and the fine-tuning stage consumes robot data for the target embodiment. Firsthand’s stated scope covers only the human side, as custom collection: first-person video, depth, 3D hand pose and one task instruction per episode, confirmed first on a Spec Pilot. Robot trajectories, retargeted actions and action labels are not Firsthand deliverables, and the site makes no claim that its data suits any H-RDT pipeline.
Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks. Billing is per validated hour.
09
H-RDT at a glance, as stated in the paper
| Item | What the paper states |
|---|---|
| Paper | H-RDT: Human Manipulation Enhanced Bimanual Robotic Manipulation (arXiv 2507.23523) |
| Authors and affiliations | Hongzhe Bi, Lingxuan Wu, Tianwei Lin, Hengkai Tan, Zhizhong Su, Hang Su, Jun Zhu; Tsinghua University and Horizon Robotics (project page) |
| Method name | H-RDT, Human to Robotics Diffusion Transformer |
| Model | Diffusion transformer, 2B parameters, flow matching |
| Stage 1 | Pretraining on the complete EgoDex dataset, 338K+ trajectories, 194 tasks, 48-dimensional human hand action |
| Stage 2 | Cross-embodiment fine-tuning with re-initialised state adaptor, action adaptor and action decoder |
| Robots named | Simulation: Aloha-Agilex-1.0 and Franka-Panda. Real world: dual-arm ARX5, Aloha-Agilex-2.0 (dual-arm Piper) and UR5 with UMI |
| Simulation benchmark | RoboTwin 2.0 |
| Code and weights | Official GitHub repository (MPL-2.0 per GitHub’s licence field) and Hugging Face model page; weights licence Not stated in the sources read |
| Firsthand relationship | None; no named model was trained on Firsthand data |
11
H-RDT beside the RDT-1B, EgoDex, EgoMimic, HAT and cross-embodiment entries
| Site entry | What the entry states | How H-RDT differs (sources read) |
|---|---|---|
| RDT-1B | Open diffusion transformer policy for bimanual manipulation, pre-trained on multi-robot data then fine-tuned on a target dual-arm robot | H-RDT says it builds on the RDT architecture but pretrains on human egocentric video with 3D hand pose before robot fine-tuning |
| EgoDex dataset | Public egocentric dexterous manipulation dataset: 829 hours, 338,000 episodes, 194 tabletop tasks, 25 tracked joints per hand | H-RDT uses the complete EgoDex dataset for stage 1; EgoDex is the authors’ pretraining source, not a Firsthand dataset |
| EgoMimic | Co-trains one policy on human embodiment data, egocentric video with 3D hand tracking from Project Aria glasses, and robot data | The H-RDT authors say EgoMimic needs paired robot data during co-training; H-RDT separates human pretraining from robot fine-tuning |
| HAT | Pairs the PH2D egocentric human-humanoid dataset with a policy whose state-action space is unified across humans and humanoids | The H-RDT authors say HAT’s retargeting assumes humanoid kinematics; H-RDT targets other morphologies through modular action adapters |
| Cross-embodiment learning | Training one policy on data from multiple embodiments, robots and human demonstrations | H-RDT pretrains on one human embodiment, then adapts to each robot; the authors contrast this with pretraining on mixed robot datasets |
12
What H-RDT consumes beside what the Firsthand site states
| Aspect | H-RDT (sources read) | What the Firsthand site states |
|---|---|---|
| Pretraining input | Egocentric human video from EgoDex with paired 3D hand pose; one egocentric camera view | Custom first-person human video, collected to a written spec |
| Hand signal | 48-dimensional vector: wrist pose (3D position, 6D orientation) and fingertip positions, both hands | 3D hand pose at 21 joints per hand with a per-joint visibility flag; no 48-dimensional vector stated |
| Depth and inertial | Not listed as pretraining inputs | Calibrated stereo depth and 6-DoF IMUs at 200 Hz |
| Language | A language instruction conditions the policy | One task instruction per episode |
| Robot data | Required for stage 2 on the target robot | Not captured; Firsthand captures first-person human video, not robot or teleoperation data |
| Training use | The authors’ own training and evaluation | No named model was trained on Firsthand data; no affiliation |
| Delivery | Public EgoDex release | Spec Pilot, then billed per validated hour; custom collection, fast and on demand |
13
Frequently asked questions
- What is H-RDT?
- H-RDT, Human to Robotics Diffusion Transformer, is a published method (arXiv 2507.23523) that pretrains a 2B-parameter diffusion transformer on egocentric human manipulation video with 3D hand pose, then fine-tunes it on robot data for bimanual manipulation. It is third-party research.
- What data does H-RDT pretrain on?
- Per the paper, the complete EgoDex dataset, described as 338K+ trajectories across 194 manipulation tasks, using a 48-dimensional human hand action made of wrist poses and fingertip positions.
- Does H-RDT need robot data?
- Yes, for stage 2. The paper fine-tunes on robot-specific data with re-initialised action encoders and decoders, so human pretraining reduces but does not remove the need for target-robot data.
- What did the authors report for H-RDT?
- They report improvements of 13.9% in simulation and 40.5% in real-world experiments over training from scratch, and a 41.6% average on a few-shot dual-arm ARX5 test against 16.0% for RDT and 31.2% for π0. These are the authors’ figures; Firsthand has not verified them.
- Is H-RDT open source?
- The authors publish an official GitHub repository, which GitHub lists under MPL-2.0, and a Hugging Face model page. The weights licence is Not stated in the sources read, so read the licence text before relying on it commercially.
- Was H-RDT trained on Firsthand data?
- No. Firsthand is not affiliated with the authors and claims no use, training or compatibility. Robot trajectories, retargeted actions, action labels and a 48-dimensional hand vector are not Firsthand deliverables.
- How can I request a custom egocentric dataset for human-to-robot learning?
- Request a custom dataset with your objects, interactions and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep, confirmed first on a Spec Pilot.
Related terms
Sources
- H-RDT: Human Manipulation Enhanced Bimanual Robotic Manipulation (arXiv 2507.23523)
- H-RDT project page (official)
- HongzheBi/H_RDT (official GitHub)
- embodiedfoundation/H-RDT (official Hugging Face model page)
- EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video (arXiv 2505.11709)
- RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation (arXiv 2410.07864)