Robot Learning From Human Video

A policy does not learn from raw pixels alone. It learns from the relationship between frames — how a hand approached an object, when contact happened, how the wrist rotated to align a grasp. Extracting that relationship requires the same streams every embodied AI pipeline needs: first-person video, aligned depth, 3D hand pose, and action labels, all captured on one clock so the timing survives into training.

• First-person (egocentric) RGB — puts the hands and manipulated objects in the frame the policy will see at inference
• Metric depth aligned to the RGB stream for reach and contact geometry
• 3D hand pose with per-joint visibility flags
• Action segments — labeled reach, grasp, and release transitions
• Motion and IMU data hardware-timestamped on the same clock as the cameras

Internet video is shot from a third-person angle, at whatever frame rate and resolution the original camera happened to use, with no depth stream and no hardware clock tying it to anything else. It can show what a task looks like from the outside. It cannot show what the task looks like from the acting viewpoint, and it cannot supply the depth or synchronized pose a manipulation policy needs to learn contact.

Yes. A policy trained only on best-case demonstrations has never seen what a recovery looks like, so it fails exactly where a clean dataset has no examples — occlusion, low light, a dropped object. Failure-case coverage matters as much as the clean demonstration itself, which is why footage is shipped with the transitions and recoveries left in rather than edited down to only the successful attempts.

A written skill spec names the tasks, environments, viewpoints, and edge cases the policy needs. Capture is then collected and validated against that spec — checking sync, calibration, coverage, and consent — before delivery in a training-ready format like RLDS or LeRobot.