Glossary
What Is ViViDex (Vision-Based Dexterous Manipulation from Human Videos)?
Last reviewed: November 2026
Short answer
ViViDex is the framework from the paper “ViViDex: Learning Vision-based Dexterous Manipulation from Human Videos” (arXiv 2404.15709, ICRA 2025). It extracts reference hand and object trajectories from human videos, trains state-based reinforcement learning policies guided by them, then distills successful rollouts into a vision-based policy for multi-fingered robot hands. Third-party research; Firsthand is not affiliated.
01
What is ViViDex, and what problem does it address?
ViViDex is introduced in ViViDex: Learning Vision-based Dexterous Manipulation from Human Videos (arXiv 2404.15709) by Zerui Chen, Shizhe Chen, Etienne Arlaud, Ivan Laptev and Cordelia Schmid. The project page lists Inria, École normale supérieure, CNRS and PSL Research University, with Ivan Laptev at Mohamed bin Zayed University of Artificial Intelligence, and says the paper was accepted at ICRA 2025. The paper expands the name as human Videos for Vision-based Dexterous manipulation. The authors say earlier work shows benefits from human videos, but performance gains were limited by noise in the estimated trajectories, and reliance on privileged object information such as ground-truth object states limits realistic use. They also say robot demonstrations collected by teleoperation demand substantial human effort and are difficult to scale. See the site’s dexterous manipulation and motion retargeting entries for the broader ideas.
02
Where does Firsthand fit next to ViViDex?
ViViDex is third-party research. Firsthand is not affiliated with the authors, and no named model, ViViDex included, was trained on Firsthand data. The paper draws its human videos from the existing DexYCB dataset. The site states custom egocentric collection to a written spec, fast and on demand, delivering first-person human video with 3D hand pose at 21 joints per hand and a per-joint visibility flag, calibrated stereo depth and 6-DoF IMUs at 200 Hz, in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license. The only language data tied to Firsthand capture is one task instruction per episode. Object trajectories, retargeted robot joint angles, reward functions or labels, state-based policy rollouts, robot demonstrations, action labels, trained policies and synthetic data are not Firsthand deliverables.
03
How does ViViDex turn human videos into a vision-based policy?
Per the paper, ViViDex has three steps. First, it extracts hand and object poses from video demonstrations, following DexMV. The hand is represented with the MANO model, which gives 3D locations of 21 joints, and an optimisation step retargets that human hand pose to the robot hand. The authors note the resulting reference trajectories look plausible but are not physically plausible.
Second, it trains a state-based policy with reinforcement learning (PPO) for each reference trajectory. The reward is split into a pre-grasp stage and a manipulation stage, and it scores how closely the robot hand and the object follow the reference. The authors also augment trajectories with new initial object positions, rotations and target positions. Third, it rolls out successful episodes from these policies and trains one vision-based policy that takes robot states and 3D point clouds from a depth camera, with no privileged object information. The point clouds are re-expressed in target, palm and fingertip coordinate frames, and the policy is trained with behaviour cloning or a 3D diffusion policy.
04
What did the authors evaluate, and what did they report?
The authors follow DexMV’s three tasks, relocate, pour and place inside, using DexYCB human demonstration videos. They focus on five objects, choosing three videos per object that are most similar to those used in DexMV, and they note that the DexMV videos are not publicly released. They also test on ten unseen objects. Simulation uses an Adroit hand in MuJoCo for comparison with prior work and an Allegro hand on a UR5 arm in SAPIEN for visual-policy training and ablations.
The abstract states that ViViDex outperforms state-of-the-art approaches on three dexterous manipulation tasks. The authors say their state-based policy uses a single video per object, where earlier methods usually needed about 100, and they report an average relocate success of 1.00 in their Table II. On a real robot they report an 88% average success rate for separately trained behaviour cloning policies, over 10 episodes per object. These are the authors’ figures on their own setups; Firsthand has not verified them.
05
What hardware, code and licence terms do the sources state?
For the real-world tests the paper names a UR5 robotic arm equipped with an Allegro hand and a single RealSense D435 RGB-D camera. Firsthand does not claim to use any of this hardware. The authors collect five robot trajectories per object by running state-based policies in simulation, then on the real robot, and train the real-world visual policy on that data.
The sources read are the arXiv paper and the official project page. Neither states a code repository or a licence for the method, so both are Not stated in the sources read, and no official GitHub or Hugging Face page is cited here. The DexYCB licence is also Not stated in the ViViDex sources read; see the site’s DexYCB entry and the official DexYCB pages before any commercial use. The authors list future work as using internet videos for more general dexterous skills and investigating advanced 3D pose estimation.
06
How does ViViDex differ from DexMV, VideoDex and MAPLE?
ViViDex builds directly on DexMV: it reuses DexMV’s trajectory extraction and tasks, then replaces demonstration-augmented reinforcement learning with a trajectory-guided reward and adds a vision-based policy. The paper cites VideoDex among works that use human videos for dexterous manipulation, but its tables compare against DexMV only. MAPLE is a different kind of method, a visual encoder rather than a policy pipeline. The first table below sets these entries side by side.
07
What does Firsthand state it can supply for this kind of work?
ViViDex consumes existing hand-object videos and produces its own trajectories, rewards and rollouts. Firsthand’s stated scope is custom collection of first-person human video, depth, 3D hand pose and one task instruction per episode, confirmed first on a Spec Pilot. The second table sets the two side by side. That is a note about format, not a compatibility claim: pose extraction, retargeting, simulation and the robot are the buyer’s own.
No off-the-shelf dataset for these tasks is stated here, so this is custom collection, fast and on demand. Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks. Billing is per validated hour.
08
ViViDex at a glance, as stated in the paper and project page
| Item | What the sources state |
|---|---|
| Paper | ViViDex: Learning Vision-based Dexterous Manipulation from Human Videos (arXiv 2404.15709) |
| Method name | ViViDex, human Videos for Vision-based Dexterous manipulation |
| Authors | Zerui Chen, Shizhe Chen, Etienne Arlaud, Ivan Laptev and Cordelia Schmid |
| Venue | ICRA 2025 (project page) |
| Human videos | DexYCB hand-object demonstration videos; three videos per object for five objects |
| Hand representation | MANO, 3D locations of 21 joints, retargeted to the robot hand |
| Tasks | Relocate, pour and place inside; five seen and ten unseen objects |
| State-based policy | Reinforcement learning (PPO) with a trajectory-guided reward, one policy per reference video |
| Visual policy | Robot states and 3D point clouds as input; behaviour cloning or 3D diffusion policy |
| Real robot named | UR5 arm with an Allegro hand and one RealSense D435 RGB-D camera |
| Code and licence | Not stated in the sources read |
| Firsthand relationship | None; no named model was trained on Firsthand data |
10
ViViDex beside related entries on this site
| Site entry | What the entry states | How ViViDex differs (sources read) |
|---|---|---|
| DexMV | Extracts 3D hand and object poses from human videos, translates them into multi-finger robot demonstrations and benchmarks reinforcement learning with demonstrations | ViViDex reuses DexMV’s extraction and tasks, but trains with a trajectory-guided reward from one video per object and distils a vision-based policy without privileged object information |
| VideoDex | Pretrains robot-hand policies on retargeted human video, then adapts with a few teleoperated demonstrations | The paper cites VideoDex but does not compare against it; ViViDex trains in simulation from reference trajectories and uses real robot data collected by the authors |
| MAPLE | Trains a visual encoder on egocentric video to predict hand contact points and 3D hand pose at contact | ViViDex trains policies, not an encoder, and uses DexYCB videos rather than egocentric video |
| Motion retargeting | Maps captured human hand and wrist motion onto a robot with different kinematics | Retargeting is the first step of ViViDex, done by optimisation against the MANO hand joints |
| Dexterous manipulation | Fine-grained, multi-finger manipulation such as regrasping, in-hand reorientation, precision pinches and tool use | ViViDex targets multi-fingered robot hands on relocate, pour and place inside |
11
What ViViDex consumes beside what the Firsthand site states
| Aspect | ViViDex (sources read) | What the Firsthand site states |
|---|---|---|
| Source video | Existing DexYCB hand-object demonstration videos | Custom first-person human video, collected to a written spec |
| Hand signal | MANO hand pose, 3D locations of 21 joints, estimated from video | 3D hand pose at 21 joints per hand with a per-joint visibility flag |
| Object signal | Object trajectories extracted from the videos | Object trajectories are not stated as a Firsthand deliverable |
| Depth | Point clouds from a depth camera during policy rollouts | Calibrated stereo depth and 6-DoF IMUs at 200 Hz |
| Task specification | Relocate, pour and place inside, defined by the authors | One task instruction per episode |
| Robot data | Simulated rollouts and five real robot trajectories per object, produced by the authors | Not delivered; Firsthand captures first-person human video, not robot or teleoperation data |
| Training use | The authors’ own training and evaluation | No named model was trained on Firsthand data; no affiliation |
| Delivery | Public paper and project page; code and licence Not stated in the sources read | Spec Pilot, then billed per validated hour; RLDS, LeRobot format or WebDataset |
12
Frequently asked questions
- What is ViViDex?
- ViViDex is a published framework (arXiv 2404.15709, ICRA 2025) that learns a vision-based policy for multi-fingered robot hands from human videos. It uses trajectories extracted from video to guide reinforcement learning, then distils the result into a visual policy. It is third-party research.
- What does the ViViDex visual policy take as input?
- Per the paper, the vision-based policy takes only robot proprioceptive states and 3D scene point clouds, so it uses no privileged object information such as ground-truth object states. The state-based policies that generate its training data do use object states.
- Which human video dataset does ViViDex use?
- The paper uses DexYCB, a dataset of human demonstration videos of hand-object interactions. It picks three videos per object for five objects. The licence is Not stated in the ViViDex sources read, so check the official DexYCB pages.
- Did the authors test ViViDex on a real robot?
- Yes. The paper names a UR5 arm with an Allegro hand and a single RealSense D435 RGB-D camera. The authors report an 88% average success rate for separately trained behaviour cloning policies; Firsthand has not verified it.
- How is ViViDex different from DexMV?
- ViViDex reuses DexMV’s trajectory extraction and tasks. The authors say it replaces task-specific reward engineering with a trajectory-guided reward, uses one video per object where DexMV used about 100, and adds a vision-based policy.
- Is ViViDex open source, and can it be used commercially?
- The paper and project page as read state no code repository or licence, so both are Not stated in the sources read. Check the official pages before any commercial use.
- Was ViViDex trained on Firsthand data?
- No. Firsthand is not affiliated with the authors and claims no use, training or compatibility. Retargeted trajectories, reward labels, robot demonstrations and policies are not Firsthand deliverables.
- How can I request a custom egocentric dataset for dexterous manipulation?
- Request a custom dataset with your objects, interactions and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep, confirmed first on a Spec Pilot.
Related terms
Sources
- ViViDex: Learning Vision-based Dexterous Manipulation from Human Videos (arXiv 2404.15709)
- ViViDex project page (official, linked from the paper)
- DexMV: Imitation Learning for Dexterous Manipulation from Human Videos (arXiv 2108.05877)
- DexYCB: A Benchmark for Capturing Hand Grasping of Objects (arXiv 2104.04631)
- DexYCB project page (official)