How to Collect Robot Pretraining Data

Pretraining data collection follows the same underlying discipline as any egocentric data collection — a written spec, calibrated and synchronized capture, validated hours, documented consent, and a standard delivery format — with two things added on top: a deliberate diversity plan across scenes, tasks, objects and people, and action labels captured alongside the video rather than added afterward. The steps below describe that methodology end to end.

The methodology matters because pretraining data is judged on how well it generalizes, not on any single demonstration. A model trained on this data has to work across scenes, tasks, objects, and people it never saw during collection, so the collection process itself has to be built around that goal from the first step rather than treating generalization as something to check for after delivery. Each of the eight steps below exists to protect one part of that outcome: the spec protects checkability, the pilot protects against scaling a flawed plan, the diversity targets protect generalization directly, calibration and sync protect the data’s usability for any model that expects geometrically consistent multi-stream input, live action labels protect label quality, QA protects against uncounted or unusable footage, consent protects the buyer’s ability to actually use and redistribute the data, and the delivery format protects against a dataset that arrives in a shape no training pipeline can ingest.

This page overlaps with how to collect egocentric video data, since egocentric human video is one of the primary sources used for pretraining. The distinction is scope: the egocentric guide covers the general capture methodology for first-person video regardless of downstream use, while this page focuses specifically on what pretraining-scale collection adds on top of that methodology — the diversity plan and the action-labeling step described in Steps 3 and 5 below.

This page describes the methodology. For a commissioned service that runs this process against a project’s pretraining spec, see pretraining data collection.

A pretraining spec names the task categories, the scene and environment types, the object set, and the participant diversity a dataset needs to cover, plus which sensor streams are required and what sync-error ceiling the data must meet. Writing this before any filming starts is what keeps the resulting dataset checkable against a standard, rather than assessed after the fact once footage already exists.

A spec that skips any of these dimensions leaves a gap that only shows up once the dataset is already in training. A task list without a scene list can still be collected entirely in one room; an object set without a participant-diversity target can still be filmed by one person’s hands. Naming every dimension up front, and treating each one as something to hit rather than something to notice missing later, is what turns the spec into a standard the delivered data can actually be checked against.

A small pilot batch is captured and validated against the spec before a full collection scales up. This surfaces problems with the spec itself, the capture tier, or the calibration setup while the cost of fixing them is still small, rather than after hundreds of hours have already been recorded against a flawed plan.

The pilot is checked the same way the full collection eventually will be: against the sync-error ceiling, against the task success and failure definitions in the spec, and against the diversity coverage targets. If the pilot fails any of those checks, that is the point where the spec, the capture rig, or the participant recruiting plan gets adjusted — not once scaled collection is already underway and the same problem is repeating across every session.

Diversity is planned into the spec as explicit coverage targets across scenes, tasks, objects and participants, not left to accumulate as a side effect of collecting more hours. A pretraining dataset that hits its hour target but repeats the same kitchen, the same few objects, and the same handful of contributors will generalize worse than a smaller, more varied one — which is the same lesson GEN-0’s official write-up draws in framing quality and diversity as mattering as much as sheer volume.

In practice this means each axis gets its own target and its own tracking during collection, not a single combined "hours collected" counter. A team can be on pace against an hours target while badly behind on scene or object coverage, and the only way to catch that during collection rather than after delivery is to track the axes separately and adjust the recruiting or scene-scheduling plan while there is still time to do so.

Cameras are calibrated for intrinsics and extrinsics, and video, depth, and pose streams are hardware-triggered off a shared clock using PTP (IEEE 1588) rather than software timestamps, which drift over a session. This is what keeps cross-stream sync error low and measurable rather than assumed from the hardware spec alone.

This step matters more for pretraining data than for a single-camera dataset because pretraining datasets typically combine several sensor streams per episode ����� video, depth, and hand or body pose — and a model that trains on misaligned streams learns spurious correlations between them instead of the real relationship. A sync-error ceiling gives a concrete, checkable number to hold every hour of footage to, rather than a subjective sense that "the cameras looked synced."

Calibration and sync also have to hold across an entire collection, not just on the day a rig is first set up. Cameras can shift slightly between sessions, and a clock source can drift if it is not re-checked, so this step is a recurring part of the methodology rather than a one-time setup cost paid before Step 1.

Hand pose, contact events, and action segments are captured live during recording rather than estimated afterward from raw footage — the same distinction EgoScale draws between its own SLAM-plus-hand-pose-estimation pseudo-labeling pipeline and directly captured labels. Labels produced this way carry less noise into training because they are not a downstream estimate of an already-recorded stream.

The practical difference shows up at the moments that matter most for a model: contact events, where a hand actually meets an object, and the transitions between one action segment and the next. A pseudo-labeling pipeline has to infer those moments from pixels after the fact, which is exactly where estimation error concentrates. Capturing them live, at the time of recording, removes that estimation step entirely.

Every hour of footage is checked against the sync-error ceiling and the capture spec before it is counted as a validated hour. Footage that fails the check is rejected and, where the spec calls for it, recaptured. Failure cases and near-misses are retained alongside successful attempts where the spec requires them, since both carry training value.

This is also where a spec written loosely in Step 1 gets exposed. If the spec did not define what counts as an acceptable attempt at a task, QA has nothing concrete to check footage against, and the "validated hour" label starts to mean whatever a reviewer decides on a given day. A spec that defines success and failure conditions per task up front is what keeps this step consistent across a full collection.

Every participant’s consent is recorded and tied to the specific footage they appear in, and faces, license plates, and screens are detected and blurred as part of the pipeline, so a buyer can trace exactly what was agreed to for any clip in the delivered dataset.

Pretraining collections tend to involve more participants and more locations than a narrow task-specific collection, simply because the diversity plan in Step 3 calls for it, which makes this record-keeping step more consequential rather than less. A dataset with hundreds of contributors across dozens of scenes needs a consent record that scales cleanly with it, not a process that was only ever built to handle a handful of participants in one studio.

Delivery uses standard robot-learning formats such as RLDS, LeRobot, WebDataset, or HDF5, with a buyer-owned, perpetual license and a documented consent chain shipping with every batch rather than being assembled after the fact.

This methodology can be commissioned against a project’s own spec rather than run independently — a fast, on-demand pretraining data collection service takes the diversity plan, capture tier, and delivery format a team needs and runs the eight steps above to produce it.

Because the spec is written first, a custom collection can be scoped around whichever gap in a team’s existing pretraining mix is actually limiting the model, rather than defaulting to a generic mixture. A team that already has broad scene and object coverage but thin coverage on a specific manipulation task can spec a collection that targets exactly that task, at the diversity level the rest of the dataset already has, instead of paying for another broad pass that duplicates coverage it already has.

The same eight-step process applies whether the resulting collection is meant to stand alone or to be blended into a larger existing pretraining corpus, since the spec, pilot, and QA steps are what make the new batch checkable against the rest of the dataset it will sit alongside.