Action-Labeled Egocentric Video for Robot Pretraining

Action-labeled egocentric video is first-person footage of a task — recorded from a head, chest, or glasses-mounted camera — paired with structured labels describing what the hands and body did during the clip: 3D hand pose, wrist trajectories, contact and grasp events, and action segments with start and end timestamps.

That is a different, more specific thing than "annotated video" in general. A caption or single classification tag per clip tells a model what the clip was broadly about; action labels tell it what happened, moment to moment, in a form a manipulation policy can train on directly.

Robot foundation models that learn manipulation through imitation learning are trained on demonstrated trajectories — sequences of observations and actions — not on video as a general visual signal. A policy’s ceiling is set by the quality and coverage of the actions it is shown, which means the action label attached to a video is what the policy actually learns from, and the video itself is context for that label rather than the training signal on its own.

This is also the practical reason egocentric pretraining corpora are described in terms of their labels, not just their hour counts. NVIDIA’s EgoScale paper describes its 20,854-hour corpus specifically as action-labeled egocentric human video, and reports its scaling results and its 54% success improvement against that labeled corpus, not against unlabeled footage of the same scale.

EgoScale derives its action labels — hand pose and related signals — from off-the-shelf SLAM and hand-pose estimation applied to the video after capture, rather than measuring them live on a purpose-built synchronized rig. That approach is what let the paper label a corpus at the tens-of-thousands-of-hours scale it reports, and it is a large part of why the resulting log-linear scaling law and 54% success improvement could be measured at all.

The tradeoff is that SLAM- and estimation-derived labels are pseudo-labels: they are inferred from the 2D or monocular signal available after the fact, not measured directly against a synchronized depth stream and hardware clock at capture time. EgoScale’s own published results demonstrate that this approach works well enough to produce a real scaling trend and a real downstream success-rate gain — the limitation is one of label provenance and precision, not of whether the approach is usable at all.

A custom egocentric collection delivers three label families as standard: 3D hand pose per hand, with a per-joint visibility flag so occluded fingers are marked rather than guessed; contact and grasp events, marking the moment of hand-object interaction; and action segments, each task broken into start- and end-timestamped spans rather than one tag for the whole clip.

Where a program spec calls for it, instance segmentation is added on top — separating the hand, the task object, and the background as distinct masks. All three label families are produced against the same synchronized depth stream and hardware clock the video itself was recorded on, which is the structural difference from labels estimated from RGB after capture.

A hand-pose or contact label is only as trustworthy as the geometry it is measured against. Cameras, depth, and IMU are calibrated together across the capture rig, and every stream shares a single hardware clock, so a 3D hand-pose annotation reflects the actual depth geometry captured at that instant rather than a position inferred from a single 2D frame taken in isolation.

This is the same requirement that separates a live-captured label from a pseudo-label produced after the fact: without synchronized depth and a shared clock, a hand-pose estimate is a best guess from appearance, not a measurement. Calibration and sync are what let an action label carry real geometric information instead of a plausible-looking approximation.

Action-labeled egocentric video is delivered in RLDS, LeRobot, WebDataset, or HDF5, matching whichever schema a training pipeline already reads, with hand pose, action segments, and contact events carried as structured fields inside that format rather than a separate file to reconcile.

JSONL is also available for annotation-only delivery when action labels are being added to video a program already has. In every format, action labels are attached at the episode and step level defined by that format’s own schema, so the labels line up with the same observation and action keys a training loop reads.

Unlabeled egocentric video is a visual record of a task; it can be watched, but a manipulation policy trained by imitation learning cannot learn a motor skill from pixels alone. Action-labeled egocentric video adds the structured signal ����� hand pose, contact, segment boundaries — that turns the same footage into something a policy can actually train on.

The practical gap shows up at pretraining scale. EgoScale’s published results are reported against a 20,854-hour corpus that is specifically action-labeled, and the paper’s log-linear scaling law and 54% success figure are both measured against that labeled data, not against raw video volume. A dataset’s usefulness for pretraining tracks its label quality and coverage at least as much as its hour count.

Start with the action vocabulary and label depth a program needs: which hand-pose, contact, and segment labels matter for the target tasks, and whether instance segmentation or de-identification should be included. That becomes a written spec, quoted for reach, timeline, and price before capture begins.

Many programs start with a Spec Pilot — a small, fast first batch that proves the label quality and action vocabulary before scaling volume — then run through the same seven-stage pipeline used for every Firsthand collection, with a median 48-hour first validated batch where coverage is deep, delivered in RLDS, LeRobot, WebDataset, or HDF5 under a buyer-owned perpetual license.