Action Segmentation for Robot Learning

An action segment is a labeled time interval marking one sub-action within an episode: a start and end timestamp with a verb and a noun, such as "grasp mug" or "open drawer". It turns a continuous clip into supervised, temporally-localized units a policy can learn from, rather than one long undifferentiated recording.

This is distinct from action recognition, which classifies an entire clip with a single label. Segmentation instead marks where each sub-action begins and ends within a longer episode, so a manipulation task like "make coffee" decomposes into a sequence of separately-timed segments: reach, grasp, pour, release.

Frame-accurate action boundaries let you train and evaluate on the moments that matter and skip the transit between them. Clip-level tags or narration text are not enough for manipulation, where the exact contact moment, not the surrounding reach or retreat, is the signal a policy needs to learn.

Segment boundaries that drift even tens of milliseconds from the true contact moment corrupt supervision exactly where it is most informative. That is why segments need to be timestamped in the same clock domain as every other stream, not annotated loosely after the fact.

Firsthand ships action_segments as {start_ns, end_ns, verb, noun} with nanosecond timestamps aligned to the same PTP clock domain as video, depth, and hand pose. Because every stream shares one clock, a segment boundary lines up with the exact frame and hand-pose sample where the sub-action actually starts or ends.

A skill spec defines the verb-noun vocabulary and the sub-actions that matter for the target task before capture starts, so segmentation is planned rather than reconstructed afterward. That same spec drives which coverage cells count as validated hours.