What Is VideoDex (Learning Dexterity from Internet Videos)?

VideoDex is named in VideoDex: Learning Dexterity from Internet Videos (arXiv 2212.04498) by Kenneth Shaw, Shikhar Bahl and Deepak Pathak of Carnegie Mellon University, accepted at CoRL 2022 according to the arXiv comments. The authors argue that collecting experience on a real robot is often not feasible because of safety, time and hardware restrictions, so they use internet videos of humans using their hands instead. Earlier work mostly used human video to pretrain only a visual representation. The authors believe much of the inefficiency comes from very large action spaces, which is worse for high degree-of-freedom robots such as dexterous hands, and that the similarity between robot and human hands makes learning from human video advantageous. For the broader ideas, see the site’s dexterous manipulation, motion retargeting and embodiment gap entries.

Firsthand is not affiliated with the VideoDex authors and makes no claim that any model, VideoDex included, was trained on Firsthand data. The paper pretrains on existing human activity video, EPIC-KITCHENS clips, which Firsthand does not supply. Firsthand captures real human egocentric video, custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license, with one task instruction per episode. Hand-pose retargeting outputs, robot trajectories, action labels, reward labels, pretrained representations, embeddings, teleoperation data, paired human-robot video, synthetic data and language annotations beyond that instruction are not Firsthand deliverables.

The authors describe a learning algorithm that uses visual, action and physical priors from human video datasets. The visual prior is an image encoder: the paper uses the publicly released ResNet-18 encoder from R3M. The action prior is a network initialization that encodes the typical hand and wrist motion for a task, learned from human video retargeted to the robot. The physical prior is a policy structure, Neural Dynamic Policies, which the authors use to produce smooth open-loop trajectories that do not overfit to noisy hand detections. The policy uses two such networks, one for the wrist and one for the hand.

Per the paper, a crop around the hand is found with OpenPose and passed to FrankMocap, which returns shape and pose parameters of the 3D MANO hand model; these are low-pass filtered. The hand pose is then mapped to a 16-degree-of-freedom robot hand by a learned retargeting network (the paper names the LEAP Hand; the project page text names the Allegro Hand).

The wrist is handled separately. The paper computes wrist pose in the camera frame with Perspective-n-Point, tracks the moving head camera with monocular SLAM (ORB-SLAM3), estimates the direction of gravity from the surface normals of floor-parallel surfaces in an estimated depth map, and rescales the trajectory into the robot workspace with a heuristic. The target arm is an xArm6. The authors state this wrist retargeting uses only 2D images from the human videos.

The authors pretrain on between 500 and 3,000 EPIC-KITCHENS clips of humans completing the same task category as the robot task, averaging 5 to 10 seconds each. The final policy is then trained on a few teleoperated demonstrations; the authors’ table lists roughly 120 to 175 robot demonstrations per task on their own setup. They report that full training takes about 10 hours on a single 2080Ti GPU. These are the authors’ figures; Firsthand has not verified them.

The authors report real-world experiments on seven tasks (pick, rotate, open, cover, uncover, place and push) on a robot arm with a dexterous hand, plus a two-finger gripper variant. On test objects they report success rates for pick of 0.77 against 0.38, 0.44 and 0.31 for three baselines, and for place of 0.80 against 0.35, 0.25 and 0.50. They conclude that visual priors matter but action priors are more impactful.

They also state limits. The method relies on off-the-shelf hand detectors that often give erroneous 6D poses when a hand is touching an object, the retargeting must be recomputed for each robot embodiment, and the policy is open-loop, so it cannot react to changes in the environment.

The first table sets VideoDex beside four site entries, summarising what those entries state. The main difference is that VideoDex pretrains an action prior from many retargeted clips of existing video and then needs a few teleoperated demonstrations, while the others start from one video, a capture system, a general mapping or pixel-space affordances.

Per the paper, it needs human video clips of the task category, hand and body detectors, camera calibration and camera-motion tracking, a robot-specific retargeting step, and a few teleoperated robot demonstrations. The authors also ablate a variant that estimates the upright direction from the camera’s own stabilization sensor, and report that their surface-normal method did better on some tasks.

Firsthand states real human egocentric capture, depth from a calibrated stereo rig, 3D hand pose, IMUs and one task instruction per episode. That is a note about format, not a compatibility claim: the retargeting, the teleoperated demonstrations and the robot are the buyer’s own. No off-the-shelf dataset for these tasks is stated here, so this is custom collection: request on-demand collection and confirm it on a Spec Pilot, with a median of 48 hours to a first validated batch where coverage is deep and three to four weeks where it is thin.

What is VideoDex?

VideoDex is a published method (arXiv 2212.04498, CoRL 2022) that uses visual, action and physical priors from internet human video to guide a robot arm and dexterous hand. It is third-party research.

What are the visual, action and physical priors in VideoDex?

Per the paper, the visual prior is a pretrained image encoder, the action prior is a policy initialization learned from human hand and wrist motion retargeted to the robot, and the physical prior is a Neural Dynamic Policy structure that gives smooth open-loop trajectories.

Does VideoDex need robot demonstrations or teleoperation?

Yes, a few. After pretraining on retargeted human video, the authors train the final policy on teleoperated demonstrations to bridge the gap between internet data and the robot embodiment.

What did the authors report for VideoDex?

They report seven real-world manipulation tasks and say VideoDex outperforms the baselines they compare, for example 0.77 on test objects for pick against 0.31 to 0.44 for three baselines. These are the authors’ figures; Firsthand has not verified them.

Was VideoDex trained on Firsthand data?

No. Firsthand is not affiliated with the authors and claims no use, training or compatibility. Retargeting outputs, robot trajectories and teleoperation data are not Firsthand deliverables.

How can I request a custom egocentric dataset for dexterous manipulation?

Request a custom dataset with your objects, interactions and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep.