Long-Horizon Task Data for Robot Pretraining

Long-horizon task data captures a job that takes several chained sub-actions to complete, rather than a single reach-grasp-release motion. "Make coffee," "assemble a kit," or "restock a shelf" are long-horizon: each is really a sequence of shorter task units — grind, pour, tamp; or scan, pick, place, seal — performed in order toward one outcome. A short-horizon demonstration, by contrast, is a single sub-action captured on its own, like "grasp mug" with nothing before or after it.

The distinction is about structure, not just duration. A long clip that repeats the same single motion is still short-horizon in structure. A long-horizon episode has internal sequence: sub-action A has to finish, often with a specific outcome (contact made, object placed), before sub-action B can meaningfully begin. That internal ordering is what a policy has to learn, and it is what a short-horizon dataset never asks it to.

This page covers what long-horizon data is, why pretraining needs it, and how action segments and episode structure make a long demonstration usable for training. For how a single sub-action is bounded and labeled, see /guides/action-segmentation-for-robot-learning; for how a job is decomposed into task units in the first place, see /guides/turning-human-jobs-into-robot-training-tasks.

Most real deployment targets are long-horizon jobs, not single motions — a warehouse pick-pack-ship sequence, a kitchen prep task, a multi-part assembly. A model pretrained only on short, isolated sub-actions has never seen how one sub-action’s outcome sets up the next, and has to learn that sequencing structure entirely during a later, smaller and more expensive training stage. Pretraining on long-horizon episodes exposes the model to that ordering at the stage where data can be collected at the widest scale, which is the same logic /guides/robot-pretraining-data lays out for pretraining priors more generally: teach the broad, transferable structure early, refine the specific embodiment and task later.

NVIDIA’s GR00T N1 research frames robot foundation model training as a mixture of data types arranged by abundance and specificity, with human video sitting at the base for its scale and viewpoint match. A long-horizon egocentric episode is still a single capture from that same base layer — the sequencing structure comes for free from the fact that a person doing a real multi-step job naturally performs it as one continuous, ordered activity, without any extra collection cost over capturing the same steps as disconnected clips.

A long-horizon episode is a single continuous recording, but it is not trained on as one undifferentiated block. Firsthand labels it with action_segments — a list of {start_ns, end_ns, verb, noun} entries, each marking one sub-action’s boundaries within the longer episode, as described in full at /guides/action-segmentation-for-robot-learning. "Make coffee" becomes reach → grasp filter → pour grounds → tamp → lock portafilter → pull shot, each with its own start and end timestamp on the same PTP clock domain as the video and depth streams.

This segmentation is what turns "one long clip" into something trainable at two levels at once: a policy can learn each sub-action’s specific motion from its segment, and separately learn the sequence and transition logic from the ordered list of segments across the episode. Neither level is available from clip-level tags or narration text, which is exactly the gap /guides/action-segmentation-for-robot-learning describes those approaches as leaving open.

A long-horizon episode uses the same canonical layout as any other Firsthand episode, documented in full at /formats/episode-schema — meta, streams, annotations, calibration — with the difference being what the annotations/action_segments list contains: several ordered entries instead of one. The episode boundary itself should match the natural start and end of the whole job (first reach to last release), not an arbitrary clip length, so the sequence a policy sees during training matches a complete real attempt at the task.

Beyond action_segments, the same annotation fields that make any Firsthand episode pretraining-ready — documented at /formats/episode-schema — carry extra weight in a long-horizon episode because they have to stay consistent across more sub-actions. hand_joints and hand_visible track hand pose per frame across the whole sequence, not just one motion; contact_events mark when the hand makes or breaks contact at each transition, which is often exactly where one sub-action ends and the next begins; and calibration/extrinsics has to hold steady across the entire episode length so a transition late in a long task is measured as precisely as one at the start.

None of these are extra fields specific to long-horizon capture — they are the same schema fields every episode ships with. What changes is that a long-horizon episode asks more of them: more segments to hold accurate boundaries on, and a longer window over which calibration and sync have to stay correct.

A long-horizon job creates a kind of failure that a short-horizon clip cannot: a mistake partway through that either gets recovered from or cascades into later steps. A dropped grip during "pour grounds" that is caught and corrected before "tamp" is a different training signal than the same drop happening during "lock portafilter" and ruining the shot. As /guides/why-failure-cases-matter-in-robot-learning describes for failure data generally, a policy that has only seen clean successful sequences has no signal for what a recoverable mistake looks like versus one that requires starting over.

Capturing this deliberately means specifying, in the skill spec, which mid-sequence failure and recovery cases to include alongside the clean successful runs — not leaving them to chance. A long-horizon dataset that only contains successful end-to-end runs is missing exactly the cases where sequencing matters most: what a policy should do when step three goes wrong.

A skill spec for a long-horizon task should name the whole job and enumerate its sub-actions in order — the same verb-noun vocabulary action segmentation will later use to label them — rather than specifying only the overall goal. /guides/how-to-write-a-skill-spec covers writing a spec in general; for a long-horizon task, the spec additionally needs to state which sub-action sequence counts as a complete, valid episode, and which mid-sequence failure and recovery variants to include as their own coverage cells.

Getting the sub-action list right in the spec up front is what lets action segmentation be planned rather than reconstructed after the fact — the same point /guides/action-segmentation-for-robot-learning makes about segmentation generally, applied here to a sequence with several steps instead of one.

The same checks that apply to any episode — calibration reprojection error, cross-stream sync, per-joint visibility flags — matter across the full length of a long-horizon episode, not just at one moment, so a review should sample transitions throughout the sequence rather than only the start.

• Segment coverage: does every sub-action in the spec have a corresponding action_segments entry, in the right order, with no gaps or overlaps?
• Transition timing: do segment boundaries land on the actual contact/release moment (checkable against contact_events), not an approximate cut?
• Sync drift over length: does calibration and timestamp alignment hold as well late in a long episode as it does at the start?
• Failure coverage: are mid-sequence failure-and-recovery cases present as their own labeled cells, not folded silently into successful runs?

A custom pretraining collection engagement captures long-horizon tasks the same way it captures any skill: against a written skill spec, with contributors performing the full multi-step job on a synchronized capture rig, and every episode delivered with action_segments, hand pose, contact events and calibration in the standard episode schema. See /custom-collection/pretraining-data-collection for how a pretraining engagement is scoped, and /guides/robot-pretraining-data for how long-horizon capture fits into the broader pretraining, mid-training and post-training picture.