What Is Phantom (Training Robots Without Robots Using Only Human Videos)?

Phantom is named in Phantom: Training Robots Without Robots Using Only Human Videos (arXiv 2503.00779) by Marion Lepert, Jiaying Fang and Jeannette Bohg of Stanford University; the project page lists CoRL 2025. The authors say data scarcity is a key challenge, that teleoperated demonstrations are difficult to scale, and that human videos are abundant and diverse but lack explicit action labels and look different from a robot. Prior methods, they write, typically co-train with robot data or use reinforcement learning. Phantom removes the robot data. For the broader ideas, see the site’s embodiment gap, motion retargeting and cross-embodiment learning entries.

Firsthand is not affiliated with the Phantom authors and makes no claim that any model, Phantom included, was trained on Firsthand data. The authors record their own third-person RGBD videos, which is a different setup from Firsthand capture. Firsthand captures real human egocentric video, custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license, with one task instruction per episode. Robot overlays, inpainted or edited video, hand-pose retargeting outputs, robot trajectories, simulated demonstrations, action labels, reward labels, teleoperation data and paired human-robot video are not Firsthand deliverables.

Per the paper, the input is human video demonstrations of a manipulation task, captured from a third-person viewpoint with an RGBD camera and performed with a pinch grasp of thumb and index finger. Each frame becomes a robot observation-action pair. A hand pose estimator labels the action. The human arm is then removed by inpainting and a rendered robot is overlaid at the estimated pose. An imitation learning policy is trained on the edited data and deployed zero-shot on a real robot, with the same virtual robot overlaid on the real robot at test time so train and test images match.

The paper states that human video has no explicit action information, so it estimates hand pose with HaMeR and refines the estimate against depth using a SAM 2 hand mask and ICP registration. To cope with occlusion it constrains the last two joints of the thumb and index finger. Each robot action is an end-effector position, a 6D orientation and a gripper opening width, taken from the thumb and index fingertips. Camera extrinsics for the target setup are assumed known.

The authors adapt the RoVi-Aug data-editing scheme from robot-to-robot transfer to the human-to-robot setting. The arm is segmented with SAM 2 and removed with E2FGVI inpainting, then a robot rendered in MuJoCo is overlaid, using depth for realistic occlusion. They compare this Hand Inpaint strategy with Hand Mask (adapted from Shadow), Red Line (adapted from EgoMimic) and an unedited Vanilla baseline. The abstract reports success rates up to 92%, and Red Line and Vanilla completed none of the in-distribution tasks. These are the authors’ results; Firsthand has not verified them and repeats no further scores.

The authors list four. Performance is bounded by hand pose estimators, which still struggle with occlusion. The robot must be able to follow the same strategy as the human, so it can collide where a hand would not. Demonstrations are limited to pinch grasps because the robots use parallel jaw grippers. Only quasi-static tasks are assessed, because the latency mismatch between demonstration and rollout is not addressed. The paper also assumes the collection camera sits at a viewpoint similar to the test camera.

The first table sets Phantom beside four site entries. The Phantom paper discusses EgoMimic and WHIRL directly and does not mention DexMV, so that row says so. The main difference is that Phantom trains on human video alone with an edited robot appearance, while the others add robot data, real-world trials, a paired simulator or a mapping step.

Per the paper, it needs human demonstration video from an RGBD camera, a hand pose estimator, segmentation and inpainting models, a renderable model of the target robot and known camera extrinsics. The second table sets this beside the Firsthand site. That is a note about format, not a compatibility claim: pose estimation, inpainting, rendering and the robot are the buyer’s own.

No off-the-shelf dataset for these tasks is stated here, so this is custom collection: request on-demand collection and confirm it on a Spec Pilot, with a median of 48 hours to a first validated batch where coverage is deep and three to four weeks where it is thin.

What is Phantom?

Phantom is a published method (arXiv 2503.00779) that converts human video demonstrations into robot observation-action pairs using hand pose estimation and visual data editing, then trains a policy with no robot data. It is third-party research.

Does Phantom need robot data?

Per the authors, no. The policy trains on edited human video only and deploys zero-shot on a real robot. The paper separately tests co-training with robot data as an extra analysis.

What does hand-to-robot data editing mean in Phantom?

Each frame has the human arm removed by inpainting and a rendered robot overlaid at the pose estimated from the hand, so the training images resemble a robot doing the task.

What limits does the Phantom paper state?

Dependence on hand pose estimators that struggle with occlusion, a robot that must follow the human’s strategy, pinch grasps only and quasi-static tasks only.

Was Phantom trained on Firsthand data?

No. Firsthand is not affiliated with the authors and claims no use, training or compatibility. Robot overlays, edited video and action labels are not Firsthand deliverables.

How can I request a custom egocentric dataset for human-video robot training?

Request a custom dataset with your tasks, environments and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep.