How Is Robot Pretraining Data Annotated?

Captured video is only pixels until it carries the structured signals a policy trains on. Annotating pretraining data means attaching those signals — where the hands are, what action is happening, when contact occurs, which pixels belong to which object — to the video and depth they describe, at the timestamp they occurred. Without that layer, egocentric footage is watchable but not trainable: a policy has no way to extract a trajectory or an action boundary from raw RGB alone.

There are two broad ways to produce that layer. One is to measure it live, during capture, from calibrated sensors on a synchronized rig — the approach described on /custom-collection/egocentric-video-annotation and /custom-collection/action-labeled-egocentric-video. The other is to estimate it after the fact from RGB alone, using computer-vision models such as SLAM and hand-pose estimation, or to infer it implicitly with a learned inverse-dynamics model — the approach EgoScale and GR00T N1 each describe in their published methodology. This guide walks through both, plus the calibration, QA, and delivery steps that sit around them.

This page is the pipeline overview. For the full wrist- and hand-pose field spec, see /guides/hand-pose-and-wrist-action-labels-for-pretraining. For the commercial programs that ship these labels on a custom collection, see /custom-collection/egocentric-video-annotation and /custom-collection/action-labeled-egocentric-video.

A pretraining-ready episode needs labels that cover trajectory, grasp shape, contact, and — where the spec calls for it — object identity, not a single caption per clip. Firsthand’s episode schema, documented in full at /formats/episode-schema, ships the following annotation fields, each tied to the synchronized streams they describe.

Not every episode needs every field — a tabletop manipulation skill spec might call for hand pose, action segments, and contact events only, while a mobile-manipulation spec adds body pose and instance segmentation. Which fields are in scope is set in the skill spec, per /guides/how-to-write-a-skill-spec.

An action segment marks where one sub-action ends and the next begins inside a longer task, rather than applying a single label to the whole clip. Each segment carries a start timestamp, an end timestamp, and a verb/noun pair �� for example pick up / knife, then place / cutting board — logged with nanosecond timestamps on the same PTP clock domain as the video and depth. That shared clock is what lets a policy line up an action boundary with the exact frame it occurred on, rather than an approximate one.

Because the verb and noun vocabulary is attached per segment rather than fixed for the whole dataset, a skill spec can define the exact action vocabulary a program needs — the terms a training stack already expects — instead of a generic taxonomy applied after the fact. See /guides/action-segmentation-for-robot-learning for more on why segment-level, rather than clip-level, labeling matters for long-horizon tasks.

Hand pose is tracked live against the captured depth stream, per hand, and stored as annotations/hand_joints — 21 joints per hand, in metres, in the head-camera frame. A per-joint annotations/hand_visible flag records occlusion explicitly, so a finger the camera could not see is marked as such rather than silently estimated through. Wrist-level motion is recorded separately as 6-DoF IMU streams (streams/imu_wrist_l, streams/imu_wrist_r) at 200 Hz, giving the coarse trajectory that hand pose’s fine joint detail sits on top of.

This combination — wrist trajectory plus per-joint hand pose — is the same structure NVIDIA’s EgoScale uses to represent action in egocentric human video, documented in its arXiv paper and on NVIDIA’s GEAR lab research page. The difference covered later in this guide is how each number was produced: tracked live against depth versus estimated after the fact from RGB. See /guides/hand-pose-and-wrist-action-labels-for-pretraining for the full field-level breakdown, and /guides/3d-hand-pose-estimation-for-robotics for the fine-manipulation case specifically.

A label is only as useful as the frame it is attached to. If a hand-joint sample and the video frame it describes were captured a few milliseconds apart, the label describes a slightly different moment than the image shows. Firsthand’s rig hardware-triggers every stream — video, depth, IMU, and hand-pose tracking — onto the same PTP clock domain, so a label and the frame it describes carry matching timestamps rather than being aligned after capture in software. See /guides/how-we-measure-sync-error for how cross-stream sync error is measured and reported.

Calibration is the other half of the same problem. hand_joints are recorded in metres in the head-camera frame, which only means what it claims to mean if the camera’s intrinsics and the SE(3) extrinsics between streams are measured accurately. That is checked per rig session against a reprojection-error threshold (rms_reproj_px < 0.28) rather than assumed to hold indefinitely from a one-time factory calibration. See /guides/camera-calibration-procedure for the calibration process itself.

A pseudo-label is an annotation produced by a model after capture, rather than measured live from calibrated sensors. NVIDIA’s EgoScale, documented on arXiv and on NVIDIA’s GEAR lab research page, derives its action labels for a 20,854-hour egocentric corpus using off-the-shelf SLAM (for camera and wrist trajectory) plus hand-pose estimation applied to RGB after the fact — not a purpose-built capture rig. NVIDIA’s GR00T N1, described in its arXiv paper, goes further for footage that has no action label at all: it uses latent actions and an inverse-dynamics model to pseudo-label unlabeled video, so footage without any ground-truth action signal can still contribute to pretraining rather than being discarded.

Neither approach is wrong for its purpose — EgoScale’s own evaluation shows its pseudo-labeled pretraining recipe transferring to real task success. The distinction matters for planning a pretraining mixture: a broad pretraining stage can tolerate pseudo-label noise better than a small, closely matched alignment stage can, which is the same two-stage structure EgoScale itself uses. See /guides/building-a-robot-pretraining-data-mixture for how to plan that split, and /guides/gr00t-n1-data-pyramid-explained for more on how GR00T N1’s data pyramid uses pseudo-labels across its layers.

For a custom collection, the labeling method is specified in the capture spec itself — hand pose and action segments tracked live against depth on a synchronized rig — rather than reverse-engineered from documentation after delivery, the way a buyer has to check for a third-party public corpus.

Annotation quality is checked in layers, at the stage where each failure mode actually occurs, rather than in a single pass at the end. Cross-stream sync error is validated against a fixed ceiling as footage is recorded. Coverage — including failure cases such as stalls, slips, and corrections — is checked against the capture spec’s required skill variations. Only footage that passes both checks counts toward the delivered total; footage that fails is logged and excluded rather than silently kept. See /guides/video-dataset-quality-assurance for the full multi-layer QA process and /guides/robot-pretraining-data-quality-checklist for the buyer-side checklist that verifies the same signals from the outside.

The underlying label fields — hand_joints, hand_visible, action_segments, contact_events, and any spec-required masks or body pose — are the same across delivery formats, but each format projects them differently. RLDS and LeRobot fold labels into the per-step observation and action structure their loaders expect. HDF5 and zarr deliver the fields closest to their native form, with per-stream nanosecond timestamps, which is why Firsthand’s own QA tooling reads that format — it is the most complete for verifying sync directly. WebDataset keys samples by frame rather than by episode, with labels delivered inline as ann.json alongside the image and depth keys for that sample.

Whichever format is chosen, the canonical field layout is /formats/episode-schema — every delivery format is a projection of it, so switching formats later does not change what a label actually contains. See /guides/pretraining-data-delivery-formats for a fuller comparison of which format fits which training stack.

To request a custom collection with a specific annotation field spec — which labels, which formats, and which QA thresholds apply — start at /custom-collection/egocentric-video-annotation, /custom-collection/action-labeled-egocentric-video, or /custom-collection/pretraining-data-collection.