What Is Zero-Shot Robot Manipulation from Passive Human Videos?

The phrase is the title of arXiv 2302.02011 by Homanga Bharadhwaj, Abhinav Gupta, Shubham Tulsiani and Vikash Kumar, listed under Carnegie Mellon University and Meta AI. The paper gives no method name, so this entry uses the full title. The authors call human videos passive because they were not collected for robot learning and have no task-level segregation, and call the framework zero-shot because there are no in-lab demonstrations or fine-tuning. It is an arXiv preprint marked as under review, so no venue is named.

Firsthand is not affiliated with the authors and makes no claim that this method, or any model, was trained on Firsthand data. Firsthand captures real human egocentric video, custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license, with one task instruction per episode. Predicted hand trajectories, robot-frame palm targets, robot trajectories, retargeted actions, action labels, reward labels, pretrained representations, embeddings, synthetic data and language annotations beyond that instruction are not Firsthand deliverables.

Per the paper, the model takes an initial image of a scene from a single viewpoint and learns a distribution over the future actions a human takes in that scene. The prediction is a 6D pose, meaning position and orientation, of a point at the centre of the human palm, over a 2 second future horizon. The authors do not model finger articulation, so the output suits a simple end-effector such as a two-finger gripper. The model is stochastic, so different samples give different trajectories.

The authors build two versions. The unconditioned model predicts plausible trajectories for a scene; the goal-conditioned model also receives a goal image and predicts trajectories that reach it.

The authors predict delta actions between time steps rather than absolute hand poses, and make the model autoregressive so each step sees the earlier actions in a trajectory. The architecture is an image transformer with an encoder-decoder design and a final C-VAE prediction head. Image features are extracted offline from the initial image, and at inference the model samples from the C-VAE prior. There is no intermediate supervision, which the authors say suits in-the-wild video.

The authors train on 2 second clips of egocentric video from Epic-Kitchens, showing everyday household activity, especially in the kitchen. For the goal-conditioned model, the last image of each clip is the goal. Ground-truth hand poses come from an off-the-shelf FrankMocap hand tracker, and only the palm-centre pose relative to the camera at the start of the window is kept.

Those poses are therefore inferred by software from ordinary video, not recorded by a sensor, so tracker errors would pass into the training signal. Epic-Kitchens is accessed under its own terms; see the site’s Epic-Kitchens entry and confirm the current licence.

The robot sees the scene through a fixed, calibrated camera, with the world frame at the robot base. The predicted delta actions are converted to absolute poses, transformed from the camera frame to the robot frame by inverse projection, and executed with an inverse kinematics controller that treats each pose as an end-effector target. The paper uses a prediction horizon of T = 7 and a Franka arm in a table-top workspace.

The authors test objects including a drawer, a door, a toaster oven, a bowl and vegetables, with 20 trials per object or goal. In their Table 1 the unconditioned model averages 50% success, against 14% for a variant trained on 30% of the data. In Table 2 the goal-conditioned model averages 37%, against 11% for the less-data variant and 12% for a 3D scene flow baseline. These are the authors’ figures on their own setup; Firsthand has not verified them.

The caveats are also theirs. The tasks are coarse, such as opening and closing drawers, pushing and tool use, in a fixed-camera table-top setup. The authors say environment-specific fine-tuning would likely improve fine-grained manipulation.

The first table sets the paper beside five site entries, summarising what those entries state. The paper itself positions against WHIRL, saying that approach still requires per-task online fine-tuning through reinforcement learning, whereas this framework uses no in-lab data at all.

Per the paper, it needs human video with hand-object interaction, a hand tracker to label poses, and a calibrated camera and controller on the robot side. Firsthand states real human egocentric capture, depth from a calibrated stereo rig, 3D hand pose at 21 joints per hand and one task instruction per episode. That is a note about format, not a compatibility claim: hand-tracking, trajectory modelling and any mapping to a robot would be work the buyer runs.

For real human demonstrations that public video does not cover, request on-demand custom collection and confirm it on a Spec Pilot, with a median of 48 hours to a first validated batch where coverage is deep and three to four weeks where it is thin.

What is zero-shot robot manipulation from passive human videos?

A published framework (arXiv 2302.02011) that predicts human hand trajectories from a scene image using human videos, then maps them to a robot with no in-lab demonstrations. It is third-party research.

What does passive mean in passive human video robot learning?

Per the authors, the videos were not collected for robot learning and have no task-level segregation. Zero-shot means no in-lab demonstrations or fine-tuning.

How does hand trajectory prediction from egocentric video drive a robot?

The model predicts a 6D palm pose trajectory over a 2 second horizon. The authors transform it into the robot frame using a calibrated camera and execute it with an inverse kinematics controller.

What did the authors report for success?

In their tables, the unconditioned model averages 50% and the goal-conditioned model 37%, with 20 trials per object or goal. These are the authors’ figures on their own setup and Firsthand has not verified them.

Was this method trained on Firsthand data?

No. Firsthand is not affiliated with the authors and claims no use, training or compatibility. Hand trajectories, action labels and pretrained representations are not Firsthand deliverables.

How can I request a custom egocentric dataset for this kind of research?

Request a custom dataset with your tasks, environments and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep.