Now recruiting capture teams worldwide. Get paid to record first-person video of real work.

Apply to capture

Glossary

What Is DexVIP (Dexterous Grasping with Human Hand Pose Priors from Video)?

DexVIP is a published method that learns robot grasping from hand poses in YouTube video. Third-party; Firsthand is not affiliated.

Last reviewed: November 2026

Short answer

DexVIP is the method in “DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Video” (arXiv 2202.00164, CoRL 2021). It curates grasp images from in-the-wild YouTube how-to videos, extracts 3D human hand poses, and uses them as a prior in a reinforcement learning reward for a simulated 30-DoF robot hand. Firsthand is not affiliated.

01

What is DexVIP, and what problem does it address?

DexVIP is named in DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Video (arXiv 2202.00164) by Priyanka Mandikal and Kristen Grauman of UT Austin and Facebook AI Research; the project page says it was accepted at CoRL 2021. The authors note that dexterous multi-fingered hands have a formidable action space, yet their morphological similarity to the human hand holds potential to accelerate robot learning. Rather than collecting human demonstrations in a lab with motion gloves, VR or teleoperation, they learn from in-the-wild YouTube videos, which they say sidesteps a more expensive and indirect way to capture human expertise.

02

Where does Firsthand fit next to DexVIP?

Firsthand is not affiliated with the DexVIP authors. No named model, DexVIP included, was trained on Firsthand data, and Firsthand does not claim to use any hardware or software named in the sources: they name the Adroit hand, the MuJoCo simulator, FrankMocap and PPO, and a dataset, HowTo100M, that Firsthand does not supply. Firsthand captures real human egocentric video, custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license, with one task instruction per episode. Grasp poses, hand pose priors, retargeted robot hand poses, grasp affordances, 3D object models, reward functions, robot trajectories, action labels, simulation environments, digital twins, co-training mixtures, pretrained representations, embeddings, synthetic data and language annotations beyond that instruction are not Firsthand deliverables.

03

How does DexVIP learn grasping from video?

Per the paper, the authors curate grasp images of 27 household objects from HowTo100M, a dataset of instructional YouTube videos, and estimate a 3D human hand pose in each image with FrankMocap. They cluster the poses per object with k-medoids and take the medoid of the largest cluster as the consensus pose, then retarget it to the Adroit hand, a 30-DoF position-controlled robot hand.

A policy is then trained with deep reinforcement learning (PPO) in the MuJoCo simulator. The reward combines grasp success, closeness to predicted object affordance regions, an entropy term and a pose term that rewards matching the consensus hand pose once the hand touches the object. The authors say the method needs only visual input from the human data, with no state-action sequences.

04

What does DexVIP take from human video, and what does it not?

The human side is a set of still grasp frames, not a labelled trajectory set. The authors report 715 curated frames, about 26 per object on average, and keep right-hand detections only because the robot is one-handed. They filter frames using HowTo100M’s weak category labels and task ids, and say the curation could be streamlined with vision methods.

Firsthand states 3D hand pose at 21 joints per hand from egocentric capture. The paper instead estimates hand pose from existing web video, and the priors, retargeted poses and affordances are the buyer’s own steps, not Firsthand deliverables.

05

What did the authors report, and what are the limits?

The authors report experiments on 27 objects with a 30-DoF simulated hand, compared with CoM, Touch, GRAFF and DAPG baselines, over 100 episodes per object and four random seeds. On the 16 ContactDB objects they report a 68% grasp success rate for the full model against 60% with affordance only, training 20% faster than GRAFF, and 64% success under heavy simulated noise. These are the authors’ simulation figures; Firsthand has not verified them.

The paper states limits: all experiments run in simulation because the authors lacked access to a real dexterous hand, so no real-robot result is reported. It handles one-handed right-hand grasps and leaves bimanual grasping to future work, it needs 3D object models, and it fails when an object’s orientation does not suit the target hand pose.

06

How does DexVIP compare with related entries on this site?

The comparison table sets DexVIP beside three site entries. The main difference is that DexVIP uses hand poses from video frames as a reward prior for reinforcement learning in simulation, where the entries it is compared with use human video as demonstrations or pretraining signal.

07

What would a DexVIP-style approach need from custom collection?

Per the sources read, it needs images of people grasping the target objects, a hand pose estimator, 3D models of those objects and a simulator with a reinforcement learning stack, all the buyer’s own. The authors use web video, and the paper does not state a need for stereo depth or IMUs. The dataset licence is Not stated in the sources read.

No off-the-shelf dataset for this is stated here, so this is custom collection: request on-demand collection and confirm it on a Spec Pilot, with a median of 48 hours to a first validated batch where coverage is deep and three to four weeks where it is thin.

08

DexVIP at a glance, as stated in the paper and project page

DexVIP at a glance, as stated in the paper and project page
ItemWhat the sources state
PaperDexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Video (arXiv 2202.00164)
AuthorsPriyanka Mandikal, Kristen Grauman
InstitutionsUT Austin; Facebook AI Research
VenueCoRL 2021, per the project page
Core ideaImpose a prior over the agent’s hand pose, taken from human grasps in YouTube video, when learning to grasp with deep reinforcement learning
Human inputCurated grasp images from in-the-wild how-to videos (HowTo100M)
Hand pose toolFrankMocap, 3D hand pose from monocular video
Robot namedAdroit 30-DoF hand, simulated in MuJoCo
Real-robot resultsNone reported; all experiments in simulation
CodeNot stated in the sources read
Dataset licenceNot stated in the sources read
Firsthand relationshipNone

10

DexVIP components, as stated in the paper

DexVIP components, as stated in the paper
ComponentWhat the sources state
Object set27 household objects with 3D models: 16 from ContactDB plus 11 others
Video frames715 curated HowTo100M frames, about 26 per object on average
Pose estimationFrankMocap 3D hand pose; right-hand detections only
Consensus posek-medoid clustering per object; medoid of the largest cluster
PolicyActor-critic network trained with PPO
RewardGrasp success, affordance contact distance, hand pose match and entropy
SupervisionVisual input only; no state-action sequences

11

DexVIP beside related entries on the site

DexVIP beside related entries on the site
Site entryWhat the entry statesHow DexVIP differs (sources read)
VideoDexCombines visual, action and physical priors from human video to guide a robot arm and dexterous hand, then adapts with a few teleoperated demonstrationsDexVIP uses hand poses from still grasp frames as a reward prior in simulation
DexMVExtracts 3D hand and object poses from human videos, translates them into multi-finger robot demonstrations and uses them for imitation learningDexVIP needs no state-action sequences; it rewards matching a consensus hand pose
Hand pose estimationRecovering 3D positions of hand and finger joints over timeDexVIP applies an existing estimator, FrankMocap, to web video frames

12

What DexVIP consumes beside what the Firsthand site states

What DexVIP consumes beside what the Firsthand site states
AspectDexVIP (sources read)What the Firsthand site states
Human videoCurated still grasp frames from web how-to videosEgocentric capture, custom collection to a written spec
Hand poseEstimated by FrankMocap from monocular frames3D hand pose at 21 joints per hand
DepthNot a stated input from the human dataDepth derived from a calibrated stereo rig
Object models3D models of 27 objects from public repositoriesNot stated as a Firsthand deliverable
LanguageNot described as an inputOne task instruction per episode
Policy and rewardReinforcement learning with a hand pose reward in simulationReward functions and simulation environments are not Firsthand deliverables
Robot-side dataA simulated 30-DoF hand; no real-robot dataRobot trajectories and retargeted robot hand poses are not Firsthand deliverables

13

Frequently asked questions

What is DexVIP?
Per the paper (arXiv 2202.00164), DexVIP learns dexterous robotic grasping from human-object interactions in in-the-wild YouTube videos by imposing a hand pose prior while training with deep reinforcement learning. It is third-party research.
What is a hand pose prior in DexVIP?
It is a target hand pose per object, the medoid of the largest cluster of 3D human hand poses estimated from curated grasp frames. A reward term favours the robot hand matching it once the hand is in contact with the object.
What video data does DexVIP use?
The authors curate 715 grasp frames from HowTo100M, about 26 per object on average, and estimate hand pose with FrankMocap. They keep right-hand detections only. The dataset licence is Not stated in the sources read.
What did the authors report for DexVIP?
In simulation on 27 objects with a 30-DoF hand, they report 68% grasp success on ContactDB objects against 60% with affordance only, and training 20% faster than GRAFF. Firsthand has not verified these figures.
Does DexVIP work on a real robot?
The paper reports no real-robot results. The authors say they lacked access to a dexterous robot hand, so all experiments are in simulation, with noise added to sensing and actuation.
Was DexVIP trained on Firsthand data?
No. Firsthand is not affiliated with the authors and no named model was trained on Firsthand data. Hand pose priors, grasp affordances, reward functions and robot trajectories are not Firsthand deliverables.
How can I request a custom egocentric dataset for dexterous grasping?
Request a custom dataset with your objects, grasps and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep.