Long-Horizon Egocentric Task Data Collection

Long-horizon egocentric task data is first-person video of a person completing a job that takes several chained sub-steps, recorded from their own point of view with synchronized sensor streams alongside. "Make coffee," "assemble a kit" and "restock a shelf" are the kind of task: each is a sequence of sub-steps in which one has to finish, often with a specific outcome, before the next can begin.

The distinction is structure rather than duration. A long clip that repeats one motion is still short-horizon in structure. A long-horizon episode has internal sequence, and it is only useful for training if that sequence is visible in the labels: which sub-steps happened, in what order, and where each one starts and ends. This page covers buying a custom collection of these demonstrations; the guide linked below explains why pretraining needs them.

A policy that must complete a whole job has to learn more than each sub-step in isolation. It has to learn the order of the steps, what a finished sub-step looks like, and when to move on. Short clips of single motions cannot teach that, because the hand-off between steps is exactly what they cut away. A full end-to-end demonstration keeps the hand-offs in the data, and the first-person view keeps the hands and the object in frame across every step.

We do not capture robot or teleoperation data, and nothing on this page claims a particular training outcome. The tradeoff to plan for is the embodiment gap between a human hand and a robot end-effector, which retargeting or pairing with robot data has to close. What a buyer gets is human first-person demonstrations in which the multi-step structure is explicit and consistent.

A long-horizon collection starts the way every custom collection does: you describe the task your pipeline needs covered, and that becomes a written skill spec. For a long task the spec does more of the work. It names the task, its sub-steps, the order they happen in, which of those are strictly sequential and which can swap, the setting, the object set, and where an episode starts and ends.

The spec also sets the capture tier, annotation depth and the accept criteria, and states how much failure and recovery coverage to include alongside clean completions. It is quoted for reach, timeline and price before any capture begins, and the accept criteria are signed into it, so acceptance is measured against what was agreed rather than judged afterwards.

Because the task is long, the episode boundary is itself a spec decision: defining what counts as a start, an end and a completed attempt up front keeps every batch comparable. The skill spec guide linked below walks through how to write one.

Paid, consented contributors perform the real task against the spec while wearing the capture rig. The instrumented tier is a head-mounted stereo rig: first-person RGB, with a metric depth stream recovered from the calibrated stereo rig rather than from a standalone depth sensor, and 6-DoF IMU data at 200 Hz from head- and wrist-mounted sensors.

Every stream is hardware-triggered onto one PTP clock domain, with int64 nanosecond timestamps per stream, so depth, motion and hand pose are aligned to the same video frame rather than approximated afterwards. The rig is calibrated before capture and again after the session ends, and the head, wrist and depth transforms are solved each time.

A full attempt is captured as one episode, in the settings this site already lists on its industry and environment pages. Collection is on demand: you do not need a standing program, and a focused brief is a normal starting point.

Captured footage is broken into the same structure used across every collection on this site. An episode groups one full attempt of the task from start to finish. Inside it, action segments divide the task into start- and end-timestamped spans, one per sub-step, rather than attaching one tag to the whole clip. In the episode schema each segment carries start_ns, end_ns, verb and noun.

Step order is therefore in the data itself: the sequence of segments inside the episode reflects the order named in the spec, so a training pipeline can learn the individual steps and the transitions between them. The labels sit alongside the other annotation fields every episode ships with, namely hand_joints and hand_visible for 3D hand pose across the whole sequence, contact_events for the moment of hand-object interaction, and objects.

None of these are extra fields specific to long-horizon capture; they are the standard episode schema fields. A long-horizon episode simply asks more of them: more segment boundaries to hold accurate, and a longer window over which pose and contact labels stay consistent.

In a long task a grasp can slip, an object can drop, or a step can need repeating, and a policy trained only on clean runs has no model of what to do next. Failure and recovery cases are part of the same labeled structure as a clean attempt, and a written spec can require them in the batch. Our failure and recovery page covers this in more depth.

Acceptance is measured against signed accept criteria written into the spec before capture begins. Each batch goes through per-batch QA, and a reject log ships with every batch. An episode is accepted only if RMS reprojection stays under 0.28 px and the drift is within tolerance; otherwise it goes to the reject log.

Only footage that clears the criteria is billed as a validated hour, and rejected items are recaptured at our cost. Many teams start with a Spec Pilot: a small, fast first batch that proves the spec and the accept criteria before you scale volume.

The buyer owns it. Every custom collection is delivered under a perpetual, non-exclusive commercial license by default, with exclusivity available, so the dataset can be trained on and productized without a research-only or unstated arrangement. Every episode is linked to a signed contributor release under a documented consent chain, and faces and screens in frame are detected and irreversibly blurred at ingest, with a reviewer confirming the result before anything ships. Clips that cannot be de-identified are discarded rather than delivered.

Delivery is in the format your stack already reads: RLDS, LeRobot, WebDataset, HDF5, zarr or Rerun .rrd, or raw streams with metadata. Episode and sub-step structure is carried in the action-segment and episode fields rather than a separate schema, and readable converter source ships alongside the data so the schema can be retargeted to your own loader.

Describe the task, its sub-steps and their order, the setting, and the failure or variation cases your pipeline is missing. That becomes a written skill spec, quoted for reach, timeline and price before capture begins, and most programs run a Spec Pilot first.

From signed spec to first validated batch, the timeline is a median of 48 hours where coverage is deep and three to four weeks where it is thin and contributors need to be recruited first. After the first batch, on-demand top-ups run against the same accept criteria, so you can add more attempts or more variation without rewriting the spec.