Glossary
What Is Hand-Object Interaction Pretraining from Videos?
Last reviewed: November 2026
Short answer
HOP, Hand-Object interaction Pretraining, is the method in “Hand-Object Interaction Pretraining from Videos” (arXiv 2409.08273) from UC Berkeley. It lifts the hand and manipulated object from in-the-wild videos into shared 3D, retargets the motion to robot actions in a simulator, and pretrains a task-agnostic base policy that is finetuned with RL or behavior cloning. Firsthand is not affiliated.
01
What is HOP, and what problem does it address?
HOP is named in Hand-Object Interaction Pretraining from Videos (arXiv 2409.08273) by Himanshu Gaurav Singh, Antonio Loquercio, Carmelo Sferrazza, Jane Wu, Haozhi Qi, Pieter Abbeel and Jitendra Malik of UC Berkeley, submitted on 12 September 2024. The project page’s BibTeX lists ICRA. The authors note that videos are abundant but only partly capture an interaction: finger force is almost impossible to estimate, and a larger embodiment gap makes human and robot actions differ. Their aim is a general manipulation prior rather than a skill tied to the video’s task.
02
Where does Firsthand fit next to HOP?
Firsthand is not affiliated with the HOP authors. No named model, HOP included, was trained on Firsthand data, and Firsthand does not claim to use any hardware or software named in the sources: they name an xArm robot with an Allegro hand, a Zed-2 stereo camera, the IsaacGym simulator, HaMeR and MCC-HO, and the DexYCB and 100 Days of Hands datasets, which Firsthand does not supply. Firsthand captures real human egocentric video, custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license, with one task instruction per episode. Robot trajectories, retargeted robot actions, action labels, hand pose priors, 3D object models, simulation environments, digital twins, co-training mixtures, pretrained representations, embeddings, synthetic data and language annotations beyond that instruction are not Firsthand deliverables.
03
How does HOP turn human videos into robot data?
Per the paper, HOP lifts the human hand and the manipulated object from in-the-wild monocular video into shared 3D, following MCC-HO with hand geometry from HaMeR. The authors skip MCC-HO’s CAD-based pose fitting, which they say reduces reconstruction quality and smoothness, and assume the camera is static.
The 3D poses go to a physics simulator. At each step the authors solve an optimization matching robot keypoints to human hand keypoints with smooth actions, then execute the action in a high-fidelity simulator, which they call simulator-in-the-loop retargeting. Object dynamics are disregarded, the scene is randomized, and trajectories are kept only if they pass retargeting-error and collision checks.
04
What video does HOP pretrain on?
Per the paper, pretraining draws on 250 right-hand-only DexYCB videos with ground-truth hand-object trajectories and about 200 videos from 100 Days of Hands, sixty percent already annotated with hand-object trajectories and the rest annotated by the authors’ 3D pipeline. The authors report about 450 videos retargeted into about 70,000 trajectories, containing neither real-world demonstrations nor complete task executions.
The HOP paper, project page and repository read do not state a dataset licence, so it is Not stated in the sources read. The site’s DexYCB entry states that DexYCB data is licensed CC BY-NC 4.0; the 100 Days of Hands licence is Not stated in the sources read.
05
How is the HOP base policy trained and finetuned?
The base policy is a GPT-2-style causal transformer that takes proprioception and observations from the past 16 timesteps and predicts the next action with an L1 loss. One base policy uses depth observations for real-world work and another uses point clouds for simulation, both trained on the same trajectories.
Finetuning uses PPO in simulation or behavior cloning on limited demonstrations on the real robot. The authors name a 7-DoF xArm with a 16-DoF Allegro hand, a Zed-2 stereo camera and the IsaacGym simulator. Firsthand does not claim to use any of them.
06
What did the authors report, and what are the limits?
On the real robot the authors test Grasp and Drop, Grasp and Pour and Grasp and Lift, with 15, 15 and 50 demonstrations and success averaged over 20 rollouts. They report single-object tasks close to saturated for most methods, while on Grasp and Lift, one policy picking four differently shaped objects, HOP scores 0.65 against 0.35 for the best baseline, a margin of 30 percentage points. In simulation they report a 2 to 5 times sample-efficiency gain over training from scratch, and the project page reports policies more than 3 times more robust to applied forces than PPO from scratch.
The paper states limits: pretraining on a single object interaction can be limiting, the camera is assumed static, and disregarded object dynamics can make motion physically implausible. These are the authors’ figures; Firsthand has not verified them.
07
How does HOP compare with related entries on this site?
The table sets HOP beside four site entries. HOP pretrains a robot policy on simulator-retargeted 3D hand-object trajectories, where the others fine-tune a visual encoder, learn latent plans, predict hand trajectories or use hand poses as a reward prior.
08
What would a HOP-style approach need from custom collection?
Per the sources read, it needs monocular videos of hands manipulating objects, a 3D hand-object reconstruction pipeline, a physics simulator and a robot model, all the buyer’s own. The paper states a static-camera assumption, while the site describes egocentric video as recorded from a camera that moves with the wearer’s head, so confirm that match first.
The paper does not state a need for stereo depth, IMUs or a specific hand-pose joint count. This is custom collection: request on-demand collection and confirm it on a Spec Pilot, with a median of 48 hours to a first validated batch where coverage is deep and three to four weeks where it is thin.
09
HOP at a glance, as stated in the paper, project page and code repository
| Item | What the sources state |
|---|---|
| Paper | Hand-Object Interaction Pretraining from Videos (arXiv 2409.08273) |
| Method name | Hand-Object interaction Pretraining (HOP) |
| Authors | Himanshu Gaurav Singh, Antonio Loquercio, Carmelo Sferrazza, Jane Wu, Haozhi Qi, Pieter Abbeel, Jitendra Malik |
| Institution | UC Berkeley |
| Venue | ICRA, per the project page BibTeX |
| Core idea | Lift hand and object to 3D, retarget to robot actions in a simulator, pretrain a base policy, finetune with RL or BC |
| Human video named | DexYCB and 100 Days of Hands subsets |
| Pretraining set (authors’ figures) | About 450 videos retargeted into about 70,000 trajectories |
| Robot named | 7-DoF xArm with a 16-DoF Allegro hand |
| Simulator | IsaacGym |
| Code | Official repository linked below |
| Dataset and code licence | Not stated in the sources read |
| Firsthand relationship | None |
11
HOP pipeline components, as stated in the paper
| Stage | What the sources state |
|---|---|
| 3D lifting | Hand and object in shared 3D from monocular video, following MCC-HO with HaMeR hand geometry; CAD fitting skipped; static camera assumed |
| Retargeting | Per-step optimization matching robot and human hand keypoints, run in a simulator; object dynamics disregarded |
| Trajectory filter | Kept only if retargeting error is below 3 cm and the arm does not collide with the table or floor (authors’ threshold) |
| Base policy | GPT-2-style causal transformer predicting the next action from the past 16 timesteps |
| Finetuning | PPO in simulation; behavior cloning on limited demonstrations in the real world |
| Real-world input | Depth images from a single stereo camera on the robot |
| Inference | Closed loop at 20 Hz |
12
HOP beside related entries on the site
| Site entry | What the entry states | How HOP differs (sources read) |
|---|---|---|
| HRP | Fine-tunes an existing visual encoder to predict hand, object and contact affordances mined from Ego4D human video | HOP trains a robot policy on simulator-retargeted trajectories rather than fine-tuning a visual encoder |
| MimicPlay | A planner learns latent plans from human play video and guides a controller trained on few teleoperated demos | HOP’s pretraining data contains no real-world demonstrations or complete task executions, per the paper |
| Zero-shot manipulation from passive human videos | Predicts human hand trajectories from a scene image and maps them to a robot | The paper cites it among hand-only priors; HOP pretrains on hand and object together |
| DexVIP | Extracts 3D hand poses from YouTube how-to videos and uses them as a prior in an RL reward for a simulated robot hand | The paper cites it among contact-pose priors; HOP embeds an implicit prior in a transformer’s weights |
13
What HOP consumes beside what the Firsthand site states
| Aspect | HOP (sources read) | What the Firsthand site states |
|---|---|---|
| Human video | Existing DexYCB and 100 Days of Hands videos, monocular | Egocentric capture, custom collection to a written spec |
| Camera | Assumed static | Camera moves with the wearer’s head |
| Depth | Not a stated pretraining input; depth is used by the real-robot policy | Depth derived from a calibrated stereo rig |
| Hand pose | Estimated from RGB with HaMeR | 3D hand pose at 21 joints per hand |
| Object | Reconstructed as a point cloud with MCC-HO | 3D object models are not Firsthand deliverables |
| Robot-side data | About 70,000 retargeted trajectories (authors’ figure) | Robot trajectories are not Firsthand deliverables |
| Language | Not a stated input | One task instruction per episode |
14
Frequently asked questions
- What is HOP?
- Per the paper (arXiv 2409.08273), HOP, Hand-Object interaction Pretraining, learns a general robot manipulation prior from 3D hand-object interaction trajectories taken from human videos. It is third-party research.
- How does HOP turn human videos into robot data?
- It lifts the hand and the object into shared 3D, retargets the hand motion to robot actions in a physics simulator, and keeps trajectories that pass error and collision checks. A transformer is then pretrained on the result.
- What data does HOP pretrain on?
- Subsets of DexYCB and 100 Days of Hands, about 450 videos retargeted into about 70,000 trajectories, per the authors. The dataset licence is Not stated in the sources read.
- What did the authors report for HOP?
- They report 0.65 against 0.35 for the best baseline on the real-robot Grasp and Lift task and a 2 to 5 times sample-efficiency gain in simulation. Firsthand has not verified these figures.
- Does a HOP-style approach need stereo depth or IMUs?
- The sources read describe monocular in-the-wild video as the pretraining input and do not state a need for stereo depth, IMUs or a specific hand-pose joint count. They state a static-camera assumption.
- Was HOP trained on Firsthand data?
- No. Firsthand is not affiliated with the authors and no named model was trained on Firsthand data. Robot trajectories, retargeted actions and 3D object models are not Firsthand deliverables.
- How can I request a custom egocentric dataset for hand-object interaction pretraining research?
- Request a custom dataset with your tasks and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep.
Related terms
Sources
- Hand-Object Interaction Pretraining from Videos (arXiv 2409.08273)
- HOP project page (official, linked from the paper)
- hgaurav2k/hop (GitHub; its README states it holds the code for the paper)
- DexYCB: A Benchmark for Capturing Hand Grasping of Objects (arXiv 2104.04631)
- DexYCB project page (official)
- Understanding Human Hands in Contact at Internet Scale (100 Days of Hands, arXiv 2006.06669)
- Reconstructing hand-held objects in 3D, MCC-HO (arXiv 2404.06507)
- geopavlakos/hamer, HaMeR (official GitHub)