Egocentric Data Collection for Imitation Learning

Imitation learning trains a policy to reproduce demonstrated behavior from example trajectories rather than from a hand-designed reward function. Behavioral cloning is the simplest form: the policy predicts the demonstrator’s action directly from their observation at each step, learned over many recorded demonstrations. Imitation learning data, then, is the set of recorded observation-action pairs — video plus the action signal that goes with it — that a policy is trained on.

Because there is no reward signal to correct for gaps in the data, a policy trained by imitation can only copy what it saw. That makes the dataset itself, not the training algorithm, the ceiling on what the policy can learn: its viewpoint, its coverage of the task distribution, and the fidelity of its action labels.

An egocentric, first-person viewpoint keeps the demonstrator’s hands and the object they are manipulating in frame together, continuously, which is the relationship an imitation-learning policy needs to learn. A third-person camera more often shows a wide scene where hands are small in frame or briefly occluded by the demonstrator’s own body — a weaker signal for a policy that has to predict a fine-grained manipulation action.

Human demonstration is also a faster route to broad task and scene coverage than a teleoperated robot and operator pair, because a person can simply perform the task directly rather than driving a robot arm through it. The tradeoff is an embodiment gap between a human hand and a robot end-effector, which retargeting or pairing has to close before the demonstration can be used to control a specific robot.

At minimum, an imitation-learning dataset needs an observation and an action at each step, aligned so the observation at time t actually corresponds to the action at time t. For an egocentric human-demonstration dataset, the observation is first-person video (and optionally depth and IMU), and the action signal is hand pose and action-segment labels tied to the same clock.

Consistent task definitions matter just as much as the signal itself: a dataset where "pick up the cup" means something slightly different from one session to the next gives a policy a noisier target to clone. That is why task definitions are fixed in a written skill spec before capture begins, rather than left to each contributor’s interpretation.

A collection starts with a written spec covering the tasks, scenes, objects, and viewpoint, plus which streams (depth, IMU) and labels (hand pose, action segments) the policy needs, and how much failure-case coverage to include alongside successful completions. The spec also sets delivery format — RLDS, LeRobot, WebDataset, HDF5, or raw streams with metadata — so the data loads directly into the training pipeline.

A Spec Pilot validates this plan on a small batch before the full order runs: a limited set of episodes captured against the written spec and reviewed with the buyer, so task definitions, label quality, and coverage can be corrected before they are repeated at volume.

The baseline stream is first-person RGB video from a head-mounted or wrist-mounted camera. Where the spec calls for it, this pairs with a synchronized depth stream, IMU motion data hardware-triggered onto the same clock as the video, and the signals an imitation-learning policy consumes directly: hand pose (3D hand keypoints through the sequence) and action-segment labels (frame-accurate labels for what the demonstrator is doing).

Every included stream is hardware-triggered onto a shared clock with per-stream timestamps, so the observation and action at a given step are actually aligned rather than approximately synchronized after the fact. Which streams and labels are included is set in the written spec — a buyer who only needs video and action segments can order just that, and a buyer training a richer policy can add depth, IMU, and hand pose.

Behavioral cloning has no reward signal to correct for gaps in the training data — the policy can only copy what it saw. If a dataset skips failure cases, transit between steps, or awkward viewpoints, the cloned policy has no way to learn how to handle them when they occur during deployment.

That is why failure and recovery cases — a grasp that slips, a step that needs repeating, a recovery back onto the task — are ordered as part of the written spec, alongside successful completions, rather than treated as footage to discard during QA.

Every episode passes through the same QA process used for any egocentric order on this site: calibration and synchronization checks on every active stream, label and annotation review for hand pose and action segments, and a pass/reject decision per episode. Episodes that fail are rejected and, where it matters to the buyer’s coverage, recaptured rather than delivered with a quality caveat. A reject log accompanies delivery.

For an imitation-learning order specifically, QA also confirms that the action-segment and hand-pose labels are correctly aligned to the observation at each step — a video and an action label that merely sit in the same episode without verified alignment do not pass the same bar as streams confirmed to line up.

Delivery format is set in the written spec before capture begins and can be RLDS, LeRobot dataset format, WebDataset, HDF5, zarr, Rerun .rrd, or raw streams with metadata — whichever structure the buyer’s training pipeline already expects, with every episode organized as an observation-action sequence.

Every contributor signs a consent release before capture, and that record travels with the episode through delivery; anonymization of bystander faces, plates, and screens is applied as a standard part of the pipeline. Delivery ships under a perpetual, buyer-owned commercial license, not a hosted-access or research-only arrangement.