Glossary
What Is a Long-Horizon Task in Robot Learning?
Last reviewed: October 2026
Short answer
A long-horizon task is a multi-step job in which several sub-actions must be completed in order toward one outcome, such as making coffee. Firsthand captures each full attempt as one episode with ordered, timestamped action segments, collected fast, on demand, to a written skill spec. Episode length is Not stated.
01
What is a long-horizon task in robot learning?
A long-horizon task is a job that takes several chained sub-actions to finish. Making coffee, assembling a kit and restocking a shelf are the site’s examples. The test is structure, not duration. A long clip that repeats one motion is still short-horizon, while a long-horizon episode has internal sequence: one sub-action finishes, often with a specific outcome, before the next can begin. A policy trained only on isolated sub-actions never sees how one step hands off to the next.
02
How is long-horizon egocentric data captured and segmented?
Firsthand records the full attempt as one episode, from first reach to final release, on a head-mounted stereo rig with 6-DoF IMU data at 200 Hz. Every stream shares one PTP clock with int64 nanosecond timestamps. Inside the episode, action_segments divide the task into {start_ns, end_ns, verb, noun} entries, one per sub-step, in time order, so step order is in the data and not in a single clip-level tag. The skill spec names the task, its sub-steps and their order, and where an episode starts and ends. Collection is fast, on demand and to that written spec.
03
Why are long-horizon tasks hard for robot policies?
A policy that must finish a whole job has to learn more than each step alone. It has to learn the order of the steps, what a finished step looks like and when to move on. Short clips cannot teach that, because the hand-off between steps is exactly what they cut away.
Mistakes also behave differently over a long task: a corrected slip is a different signal from one that cascades into later steps, so a spec can require failure and recovery cases.
NVIDIA’s GR00T N1 paper (arXiv 2503.14734, Section 4.6) states that the model focuses primarily on short-horizon tabletop manipulation tasks and names long-horizon loco-manipulation as future work. Firsthand does not state any robot results from its own data: Not stated.
04
What labels make multi-step task data usable for training?
The first table lists the fields that carry the structure. None is specific to long-horizon capture; they are the standard episode schema fields. Action segments mark each sub-step’s boundaries. hand_joints and hand_visible track 3D hand pose across the whole sequence, and contact_events mark contact and release, which often coincide with a segment transition.
A long episode asks more of these fields: more boundaries to hold accurate, and a longer window for pose, contact and sync to stay consistent. An episode is accepted only if RMS reprojection stays under 0.28 px.
05
How does a long-horizon task differ from task decomposition and action chunking?
They sit at different levels. A long-horizon task is a property of the job: it has several ordered sub-actions. Task decomposition is the method that splits a job into short, repeatable task units, each with a clear start, end, outcome and object set.
Action chunking is a model technique in which a policy predicts a block of consecutive future actions at once. Firsthand does not deliver action chunks and states no chunk length. A buyer training a chunked policy builds it on top of the delivered labels.
06
How do you request a custom long-horizon dataset?
Write down the job, its sub-steps in order, the setting, the object set, where an attempt starts and ends, and how much failure and recovery coverage you need. Choose a delivery format: RLDS, LeRobot, WebDataset, HDF5 / zarr or Rerun RRD. A custom collection starts from a written skill spec, typically proven on a smaller Spec Pilot, with a first validated batch in a median of 48 hours where coverage is deep. Anything marked Not stated below is set per program in the spec.
07
Long-horizon-related fields Firsthand states, with delivery names and formats
| Field | What it encodes | Delivery field | Stated shape |
|---|---|---|---|
| Action segments | One entry per sub-action, in time order | annotations/action_segments | {start_ns, end_ns, verb, noun}; 97 verbs, 512 nouns |
| Episode duration | Spans the full multi-step job | meta.duration_ns | Per episode; typical length Not stated |
| Hand joints | 3D hand pose across the whole sequence | annotations/hand_joints | float32 [T, 2, 21, 3], metres, head-camera frame |
| Joint visibility | Whether each joint is visible | annotations/hand_visible | Per joint; shape Not stated |
| Contact events | Contact and release moments | annotations/contact_events | Shape Not stated |
| Timestamps | Per-stream time on one clock | *_timestamp_ns | int64 nanoseconds, PTP domain |
09
Long-horizon task compared with related terms
| Term | What it is | Level | Firsthand relationship |
|---|---|---|---|
| Long-horizon task | A job with several ordered sub-actions toward one outcome | The job | Captured as one episode with ordered action segments |
| Task decomposition | Method that splits a job into short task units | Method | Each chosen unit becomes a skill spec |
| Action segment | Labeled interval of one sub-action | Label inside an episode | Delivered as {start_ns, end_ns, verb, noun} |
| Action chunking | Policy predicts a block of consecutive future actions | Model technique | Not delivered; no chunk length or chunked export stated |
10
What the site states and what is Not stated
| Detail | Stated on the site | Status |
|---|---|---|
| Episode length or duration range | No | Not stated; set per program |
| Number of sub-steps or segments per episode | No | Not stated; set per program |
| Threshold that makes a task long-horizon | No | Not stated |
| Nested or hierarchical segment labels | No | Not stated; the site describes one ordered list |
| Success or completion labels | No | Not stated |
| Share of failure and recovery episodes | No | Not stated; set in the spec |
| Robot success rates from this data | No | Not stated |
11
Frequently asked questions
- What is a long-horizon task?
- A job that takes several chained sub-actions, performed in order toward one outcome, such as making coffee or restocking a shelf. It is defined by sequence, not length.
- How long is a long-horizon episode?
- Not stated. The site gives no episode length, step count or threshold. An episode spans the full job, from first reach to final release.
- Is a long video clip a long-horizon task?
- Not by itself. A long clip that repeats one motion is still short-horizon in structure. Long-horizon data has internal sequence that is visible in the labels.
- How is a long-horizon task labeled?
- Each full attempt is one episode, divided into action segments with start and end timestamps, a verb and a noun, alongside 3D hand pose, joint visibility and contact events.
- Is a long-horizon task the same as task decomposition?
- No. A long-horizon task is the multi-step job itself. Task decomposition is the method that splits a job into short, repeatable task units.
- Do failures and recoveries belong in long-horizon data?
- Yes, where a policy needs them. A written spec can require mid-sequence failure and recovery cases. How many to include is Not stated and set in the spec.
- How can I order a custom long-horizon dataset quickly?
- Request a custom dataset with your job, sub-steps in order and delivery format. Collection is on demand against a written spec, with a first validated batch in a median of 48 hours where coverage is deep.