What Is DexMV (Imitation Learning for Dexterous Manipulation from Human Videos)?

DexMV is named in DexMV: Imitation Learning for Dexterous Manipulation from Human Videos (arXiv 2108.05877) by Yuzhe Qin, Yueh-Hua Wu, Shaowei Liu, Hanwen Jiang, Ruihan Yang, Yang Fu and Xiaolong Wang of UC San Diego; the project page and the authors’ repository place it at ECCV 2022. The authors note that computer vision has made progress on hand-object interaction, yet it is still very challenging for robots to perform complex dexterous manipulation. Reinforcement learning alone needs a large amount of training data for a multi-finger hand, and earlier demonstrations were collected with a VR headset and a motion capture glove, which the authors call relatively high-cost and not scalable. Their answer is to record human hands on video instead. For the broader ideas, see the site’s dexterous manipulation, motion retargeting and hand-object interaction entries.

Firsthand is not affiliated with the DexMV authors and makes no claim that any model, DexMV included, was trained on Firsthand data. The authors record their own videos in a fixed two-camera frame, which is a different setup from Firsthand capture. Firsthand captures real human egocentric video, custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license, with one task instruction per episode. Object 6-DoF pose labels, hand-pose retargeting outputs, robot trajectories, simulated demonstrations, action labels, reward labels, teleoperation data and paired human-robot video are not Firsthand deliverables.

Per the paper, the platform pairs a computer vision system with a simulation system, both set up for the same tasks. The vision system is a cubic frame with two RealSense D435 RGBD cameras, one at the top front and one at the top left, so a person performs the task inside the frame and is recorded from a front and a side view. The simulation system is built on MuJoCo with the Adroit multi-finger hand, which the paper describes as having 30 degrees of freedom. The three tasks are relocate, pour and place inside, using YCB objects, and the authors report collecting 100 demonstrations per object for relocate and 100 for each of the other two tasks.

The pipeline has three stages. First, 3D hand and object poses are extracted from the videos. Objects get a 6-DoF pose from a model trained on the YCB dataset; the hand is represented with the MANO model and refined by optimization against 2D joints and depth, using two cameras by default to handle occlusion. Second, demonstration translation converts the human trajectory into robot data: hand motion retargeting maps the human hand to the robot hand, then robot action estimation recovers motor commands from the retargeted joint angles using inverse dynamics on a smooth, minimum-jerk trajectory. The authors also align the video, around 30 Hz, to the simulator, which runs at 120 Hz.

The authors emphasise that retargeting alone is not demonstration generation. Their repository notes that generating demonstrations also involves trajectory generation, time alignment, inverse dynamics and hindsight.

DexMV does not rely on behavior cloning. The authors argue that video demonstrations may not be optimal for perfect imitation, but that a diverse set helps augment reinforcement learning, which can learn from both successful and unsuccessful trials. They benchmark algorithms that add demonstrations to reinforcement learning (GAIL+, DAPG and the state-only SOIL) against a TRPO reinforcement learning baseline. The abstract states that the demonstrations improve robot learning by a large margin and solve complex tasks that reinforcement learning alone cannot solve. Those are the authors’ results; Firsthand has not verified them and repeats no scores here.

The first table sets DexMV beside four site entries, summarising what those entries state. The main difference is that DexMV starts from videos the authors record themselves for tasks that also exist in a paired simulator, and then trains in simulation, while the others start from internet video, a capture system, one third-person video or a general mapping.

Per the paper, it needs human videos of the task, depth from calibrated cameras, hand and object pose estimators, a simulated copy of each task with a robot hand model, and a retargeting step specific to that hand. The second table sets this beside the Firsthand site. That is a note about format, not a compatibility claim: pose estimation, retargeting, simulation and the robot are the buyer’s own.

No off-the-shelf dataset for these tasks is stated here, so this is custom collection: request on-demand collection and confirm it on a Spec Pilot, with a median of 48 hours to a first validated batch where coverage is deep and three to four weeks where it is thin.

What is DexMV?

DexMV is a published platform and pipeline (arXiv 2108.05877, ECCV 2022) that extracts 3D hand and object poses from human videos, translates them into multi-finger robot demonstrations and uses them for imitation learning. It is third-party research.

What is demonstration translation in DexMV?

Per the paper, it converts a human hand-object trajectory into robot demonstrations in two steps: hand motion retargeting to robot joint angles, then robot action estimation to recover the commands needed for learning.

Does DexMV need real robot data?

The paper trains and tests policies in a simulation of the tasks with a multi-finger robot hand, using demonstrations translated from human video. Anything beyond that is for the authors’ paper and repository to say.

What did the authors report for DexMV?

The abstract states that the demonstrations improve robot learning by a large margin and solve complex tasks that reinforcement learning alone cannot solve. These are the authors’ claims; Firsthand has not verified them.

Was DexMV trained on Firsthand data?

No. Firsthand is not affiliated with the authors and claims no use, training or compatibility. Retargeting outputs, robot trajectories and simulated demonstrations are not Firsthand deliverables.

How can I request a custom egocentric dataset for dexterous manipulation?

Request a custom dataset with your objects, interactions and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep.