Skill-Level vs Frame-Level Annotations for Robot Learning

The difference is temporal granularity: a skill-level annotation covers a whole sub-action, and a frame-level annotation covers a single instant. An action segment — the skill-level unit — is one {start_ns, end_ns, verb, noun} record that spans however many frames the sub-action takes, from the first reach to the final release. A frame-level annotation, like a hand-joint position or an instance mask, is a separate value attached to every individual frame within that span, whether or not anything about the label changes from one frame to the next.

Neither level replaces the other; they answer different questions a policy needs answered. Skill-level labels answer "what sub-action is happening, and when does it start and end" — the structure a task decomposes into. Frame-level labels answer "what does the scene look like right now" — where the fingers are, whether contact has occurred, which pixels belong to which object, at this exact timestamp. A policy trained only on skill-level labels knows the boundaries of "grasp mug" but not how the hand was shaped while grasping it; a policy trained only on frame-level labels has dense per-frame detail but no structured notion of where one sub-action ends and the next begins.

A skill-level label is an action segment: a labeled time interval marking one sub-action within an episode, with a start timestamp, an end timestamp, a verb, and a noun — for example "grasp mug" or "open drawer". It turns a continuous, undifferentiated clip into a sequence of supervised, temporally-localized units. A longer task like "make coffee" decomposes into a sequence of separately-timed segments: reach, grasp, pour, release, each with its own boundaries rather than one label covering the whole clip.

This is distinct from action recognition, which classifies an entire clip with a single label and has no internal structure. It is also distinct from a narration or caption, which describes what happened in free text without machine-parseable timestamps. Action segmentation is deliberately narrower and more structured than either: a fixed verb-noun vocabulary, exact start and end times, and nothing else. Firsthand ships action_segments with nanosecond timestamps aligned to the same PTP clock domain as video, depth, and hand pose, so a segment boundary lines up with the exact frame where the sub-action actually starts or ends, not an approximation reconstructed after the fact.

A frame-level label is a value attached to a single timestep rather than to a span of time. Firsthand’s episode schema carries several: hand_joints, the per-joint 3D position of the fingers and hand (float32 [T, 2, 21, 3], metres, head-camera frame), tracked live against depth on every frame; hand_visible, a per-joint visibility flag recorded alongside hand_joints so an occluded finger is marked rather than silently guessed; contact_events, recorded alongside hand pose and depth during capture to flag when and where a hand made or broke contact with an object; and instance_masks (uint16 [T, 480, 848]), which separate objects and body parts pixel-by-pixel on each frame where the program spec calls for them.

What makes these frame-level rather than skill-level is that each one is recorded independently at every timestep, with no assumption that the value stays constant across a whole sub-action. A hand can be visible for part of a "grasp mug" segment and occluded for another part of the same segment; hand_visible captures that frame by frame, while the action_segment covering "grasp mug" is a single record spanning both.

Pretraining, mid-training, and post-training sit at very different scales, and the annotation level that matters most shifts with them. Pretraining is broad and human-sourced at large scale, where the priority is coverage across scenes, tasks, and people; at that scale, skill-level action segments plus frame-level wrist and hand-pose fields together supply an action representation without a robot present, since human footage has no robot joint-angle log to fall back on. Mid-training is a much smaller dataset closely matched to the deployment robot and viewpoint, where frame-level precision — hand pose, contact, occlusion flags — matters more per episode because there are far fewer episodes to average errors out over. Post-training (fine-tuning) narrows further to specific tasks and one target robot, where the skill-level action segments define exactly which sub-actions the policy is being evaluated against.

In practice, both levels are useful at every stage; what changes is which one a given spec should prioritize checking first. A pretraining spec that only defines a verb-noun vocabulary but never checks hand-pose occlusion handling will still get some benefit from segment structure alone, but will miss the fine-manipulation signal that only frame-level hand pose supplies. A narrow post-training spec that only checks frame-level hand pose but never defines segment boundaries makes it hard to evaluate whether the policy performs the right sub-action at the right time, as opposed to just moving plausibly.

Within one episode, skill-level and frame-level labels are not alternatives — they are two layers describing the same span of time at different resolutions, tied together by a shared clock. An action_segment record spans, say, 2.4 seconds of a "grasp mug" sub-action. Within that span, every frame carries its own hand_joints, hand_visible, and contact_events values, sampled at whatever rate the stream runs. Because every stream, including the action segments and the frame-level annotations, is hardware-triggered onto the same PTP clock domain, a query for "frame-level state during this skill-level segment" is a straightforward timestamp lookup rather than a best-effort alignment done after the fact.

This layering is what lets one episode support very different downstream uses without re-annotation. A policy that only needs to learn segment structure can train on action_segments alone. A policy that needs fine manipulation detail can pull hand_joints and contact_events for the frames inside a given segment. A policy that needs both — what sub-action, and exactly how the hand was shaped while performing it — reads both layers off the same episode, aligned to the same nanosecond timestamps, rather than needing a second annotation pass to reconcile them.

Skill-level and frame-level annotations fail in different ways, so they are checked differently. A skill-level failure is a wrong or missing boundary: a segment that starts or ends off by too many frames from the true contact moment, or a sub-action that was never segmented at all. Because segments are timestamped in the same clock domain as video and depth, a boundary check can be automated: verify the segment’s start_ns and end_ns line up with an actual change in hand pose or contact state at that timestamp, rather than drifting into the surrounding reach or retreat.

A frame-level failure is a wrong or missing per-frame value: an occluded finger reported as visible, a hand-joint position that does not match the depth geometry at that instant, or a mask boundary that does not track the object it is meant to cover. Firsthand checks frame-level geometry against calibration — hand_joints are recorded in metres in the head-camera frame, which only means what it claims if intrinsics and extrinsics are measured accurately, so each rig session is checked against a reprojection-error threshold (rms_reproj_px < 0.28) before its frame-level labels are considered trustworthy. Cross-stream sync error, the residual timing offset between simultaneously captured streams, is measured the same way regardless of which annotation level depends on it, because both levels are only as good as the clock alignment underneath them.

A skill spec is the written contract for a capture program: the named skill, the environments and conditions to cover, the acceptance criteria, and the target hours, agreed before a single contributor is briefed. Annotation level belongs in that same spec, stated explicitly rather than left implied. For the skill-level layer, that means naming the verb-noun vocabulary and which sub-actions within a task should get their own segment boundary, since a generic taxonomy will not match a specific target task. For the frame-level layer, that means stating which fields are required — hand_joints and hand_visible at minimum for any task involving grasping, contact_events where contact timing matters, instance_masks only where the downstream policy actually needs pixel-level separation.

To request a custom collection with a specific skill-level and frame-level annotation spec, start at /custom-collection/egocentric-video-annotation or /custom-collection/pretraining-data-collection.

Being explicit about both levels in the spec also sets expectations for delivery format. RLDS, LeRobot, WebDataset, HDF5, and zarr all carry the same underlying fields — action_segments at the skill level, hand_joints, hand_visible, contact_events, and instance_masks at the frame level — projected into each format’s own structure. The canonical field layout for both levels is documented at /formats/episode-schema; every delivery format is a projection of it, so a spec written against that schema does not need to change if the delivery format changes later.