Glossary
What Is DexVIP (Dexterous Grasping with Human Hand Pose Priors from Video)?
Last reviewed: November 2026
Short answer
DexVIP is the method in “DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Video” (arXiv 2202.00164, CoRL 2021). It curates grasp images from in-the-wild YouTube how-to videos, extracts 3D human hand poses, and uses them as a prior in a reinforcement learning reward for a simulated 30-DoF robot hand. Firsthand is not affiliated.
01
What is DexVIP, and what problem does it address?
DexVIP is named in DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Video (arXiv 2202.00164) by Priyanka Mandikal and Kristen Grauman of UT Austin and Facebook AI Research; the project page says it was accepted at CoRL 2021. The authors note that dexterous multi-fingered hands have a formidable action space, yet their morphological similarity to the human hand holds potential to accelerate robot learning. Rather than collecting human demonstrations in a lab with motion gloves, VR or teleoperation, they learn from in-the-wild YouTube videos, which they say sidesteps a more expensive and indirect way to capture human expertise.
02
Where does Firsthand fit next to DexVIP?
Firsthand is not affiliated with the DexVIP authors. No named model, DexVIP included, was trained on Firsthand data, and Firsthand does not claim to use any hardware or software named in the sources: they name the Adroit hand, the MuJoCo simulator, FrankMocap and PPO, and a dataset, HowTo100M, that Firsthand does not supply. Firsthand captures real human egocentric video, custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license, with one task instruction per episode. Grasp poses, hand pose priors, retargeted robot hand poses, grasp affordances, 3D object models, reward functions, robot trajectories, action labels, simulation environments, digital twins, co-training mixtures, pretrained representations, embeddings, synthetic data and language annotations beyond that instruction are not Firsthand deliverables.
03
How does DexVIP learn grasping from video?
Per the paper, the authors curate grasp images of 27 household objects from HowTo100M, a dataset of instructional YouTube videos, and estimate a 3D human hand pose in each image with FrankMocap. They cluster the poses per object with k-medoids and take the medoid of the largest cluster as the consensus pose, then retarget it to the Adroit hand, a 30-DoF position-controlled robot hand.
A policy is then trained with deep reinforcement learning (PPO) in the MuJoCo simulator. The reward combines grasp success, closeness to predicted object affordance regions, an entropy term and a pose term that rewards matching the consensus hand pose once the hand touches the object. The authors say the method needs only visual input from the human data, with no state-action sequences.
04
What does DexVIP take from human video, and what does it not?
The human side is a set of still grasp frames, not a labelled trajectory set. The authors report 715 curated frames, about 26 per object on average, and keep right-hand detections only because the robot is one-handed. They filter frames using HowTo100M’s weak category labels and task ids, and say the curation could be streamlined with vision methods.
Firsthand states 3D hand pose at 21 joints per hand from egocentric capture. The paper instead estimates hand pose from existing web video, and the priors, retargeted poses and affordances are the buyer’s own steps, not Firsthand deliverables.
05
What did the authors report, and what are the limits?
The authors report experiments on 27 objects with a 30-DoF simulated hand, compared with CoM, Touch, GRAFF and DAPG baselines, over 100 episodes per object and four random seeds. On the 16 ContactDB objects they report a 68% grasp success rate for the full model against 60% with affordance only, training 20% faster than GRAFF, and 64% success under heavy simulated noise. These are the authors’ simulation figures; Firsthand has not verified them.
The paper states limits: all experiments run in simulation because the authors lacked access to a real dexterous hand, so no real-robot result is reported. It handles one-handed right-hand grasps and leaves bimanual grasping to future work, it needs 3D object models, and it fails when an object’s orientation does not suit the target hand pose.
06
How does DexVIP compare with related entries on this site?
The comparison table sets DexVIP beside three site entries. The main difference is that DexVIP uses hand poses from video frames as a reward prior for reinforcement learning in simulation, where the entries it is compared with use human video as demonstrations or pretraining signal.
07
What would a DexVIP-style approach need from custom collection?
Per the sources read, it needs images of people grasping the target objects, a hand pose estimator, 3D models of those objects and a simulator with a reinforcement learning stack, all the buyer’s own. The authors use web video, and the paper does not state a need for stereo depth or IMUs. The dataset licence is Not stated in the sources read.
No off-the-shelf dataset for this is stated here, so this is custom collection: request on-demand collection and confirm it on a Spec Pilot, with a median of 48 hours to a first validated batch where coverage is deep and three to four weeks where it is thin.
08
DexVIP at a glance, as stated in the paper and project page
| Item | What the sources state |
|---|---|
| Paper | DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Video (arXiv 2202.00164) |
| Authors | Priyanka Mandikal, Kristen Grauman |
| Institutions | UT Austin; Facebook AI Research |
| Venue | CoRL 2021, per the project page |
| Core idea | Impose a prior over the agent’s hand pose, taken from human grasps in YouTube video, when learning to grasp with deep reinforcement learning |
| Human input | Curated grasp images from in-the-wild how-to videos (HowTo100M) |
| Hand pose tool | FrankMocap, 3D hand pose from monocular video |
| Robot named | Adroit 30-DoF hand, simulated in MuJoCo |
| Real-robot results | None reported; all experiments in simulation |
| Code | Not stated in the sources read |
| Dataset licence | Not stated in the sources read |
| Firsthand relationship | None |
10
DexVIP components, as stated in the paper
| Component | What the sources state |
|---|---|
| Object set | 27 household objects with 3D models: 16 from ContactDB plus 11 others |
| Video frames | 715 curated HowTo100M frames, about 26 per object on average |
| Pose estimation | FrankMocap 3D hand pose; right-hand detections only |
| Consensus pose | k-medoid clustering per object; medoid of the largest cluster |
| Policy | Actor-critic network trained with PPO |
| Reward | Grasp success, affordance contact distance, hand pose match and entropy |
| Supervision | Visual input only; no state-action sequences |
11
DexVIP beside related entries on the site
| Site entry | What the entry states | How DexVIP differs (sources read) |
|---|---|---|
| VideoDex | Combines visual, action and physical priors from human video to guide a robot arm and dexterous hand, then adapts with a few teleoperated demonstrations | DexVIP uses hand poses from still grasp frames as a reward prior in simulation |
| DexMV | Extracts 3D hand and object poses from human videos, translates them into multi-finger robot demonstrations and uses them for imitation learning | DexVIP needs no state-action sequences; it rewards matching a consensus hand pose |
| Hand pose estimation | Recovering 3D positions of hand and finger joints over time | DexVIP applies an existing estimator, FrankMocap, to web video frames |
12
What DexVIP consumes beside what the Firsthand site states
| Aspect | DexVIP (sources read) | What the Firsthand site states |
|---|---|---|
| Human video | Curated still grasp frames from web how-to videos | Egocentric capture, custom collection to a written spec |
| Hand pose | Estimated by FrankMocap from monocular frames | 3D hand pose at 21 joints per hand |
| Depth | Not a stated input from the human data | Depth derived from a calibrated stereo rig |
| Object models | 3D models of 27 objects from public repositories | Not stated as a Firsthand deliverable |
| Language | Not described as an input | One task instruction per episode |
| Policy and reward | Reinforcement learning with a hand pose reward in simulation | Reward functions and simulation environments are not Firsthand deliverables |
| Robot-side data | A simulated 30-DoF hand; no real-robot data | Robot trajectories and retargeted robot hand poses are not Firsthand deliverables |
13
Frequently asked questions
- What is DexVIP?
- Per the paper (arXiv 2202.00164), DexVIP learns dexterous robotic grasping from human-object interactions in in-the-wild YouTube videos by imposing a hand pose prior while training with deep reinforcement learning. It is third-party research.
- What is a hand pose prior in DexVIP?
- It is a target hand pose per object, the medoid of the largest cluster of 3D human hand poses estimated from curated grasp frames. A reward term favours the robot hand matching it once the hand is in contact with the object.
- What video data does DexVIP use?
- The authors curate 715 grasp frames from HowTo100M, about 26 per object on average, and estimate hand pose with FrankMocap. They keep right-hand detections only. The dataset licence is Not stated in the sources read.
- What did the authors report for DexVIP?
- In simulation on 27 objects with a 30-DoF hand, they report 68% grasp success on ContactDB objects against 60% with affordance only, and training 20% faster than GRAFF. Firsthand has not verified these figures.
- Does DexVIP work on a real robot?
- The paper reports no real-robot results. The authors say they lacked access to a dexterous robot hand, so all experiments are in simulation, with noise added to sensing and actuation.
- Was DexVIP trained on Firsthand data?
- No. Firsthand is not affiliated with the authors and no named model was trained on Firsthand data. Hand pose priors, grasp affordances, reward functions and robot trajectories are not Firsthand deliverables.
- How can I request a custom egocentric dataset for dexterous grasping?
- Request a custom dataset with your objects, grasps and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep.
Related terms
- DexMV (Imitation Learning for Dexterous Manipulation from Human Videos)
- VideoDex (Learning Dexterity from Internet Videos)
- ViViDex (Vision-Based Dexterous Manipulation from Human Videos)
- Hand pose estimation
- Hand-object interaction (HOI)
- Dexterous manipulation
- HuDOR (Bridging the Human to Robot Dexterity Gap through Object-Oriented Rewards)
Sources
- DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Video (arXiv 2202.00164)
- DexVIP project page (official, linked from the paper)
- HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (arXiv 1906.03327)
- FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integration (arXiv 2008.08324)
- Dexterous Robotic Grasping with Object-Centric Visual Affordances, GRAFF (arXiv 2009.01439)
- DexMV: Imitation Learning for Dexterous Manipulation from Human Videos (arXiv 2108.05877)
- VideoDex: Learning Dexterity from Internet Videos (arXiv 2212.04498)