How to Evaluate a Robot Pretraining Data Pilot Before Scaling

A pretraining data pilot is a small, fast first batch of egocentric footage captured against a written skill spec, sized for pretraining-scale coverage rather than a single fine-tuning task. Firsthand calls this a Spec Pilot: a buyer writes the skill spec — the task, geometry, condition cells, and reject rule — and the pilot is captured against that exact document, at low volume, before any commitment to full-volume hours.

At pretraining scale, a pilot is not asking whether one task works on one rig. It is asking whether the spec, the accept criteria, and the capture process will hold up once the same pipeline is run across many scenes, tasks, objects, and contributors. A pilot that passes on a narrow slice of footage but cannot show it would hold at scale has not actually answered that question.

See pilot data collection for the general definition and the pilot-to-production comparison table.

Committing full-volume budget to a pretraining corpus on the strength of a pitch deck is a bet on an unproven process. A pilot turns that bet into a measured result: the real reject rate against the written reject rule, the real coverage fill against the spec’s condition cells, and the real sync and calibration numbers on actual footage, rather than numbers from a demo reel.

The value of evaluating the pilot carefully is that scaling from a pilot to full volume is not supposed to introduce new process risk. The spec, the accept criteria, and the billing unit all carry over unchanged from pilot to production. If the pilot evaluation is thorough, the only variable that changes at scale is the number of validated hours delivered — not the process that delivers them.

Because a pretraining corpus is judged on aggregate coverage rather than any single clip, a pilot that looks clean clip-by-clip can still fail the real test if its diversity is narrow. Evaluating a pilot is therefore as much about checking the corpus-level coverage report as it is about spot-checking individual footage.

The first check is literal conformance to the written skill spec: does the delivered footage show the named task, by verb and noun, with the geometry the spec required to stay in frame? A skill spec is the written contract the whole collection is measured against, and a pilot that drifts from it — a different task variant, a camera angle that cuts off the required geometry, a condition cell captured informally instead of against its written description — has not actually tested the spec.

Spec conformance is checked against the accept criteria and reject rule written into the spec before capture started, not against a general sense of whether the footage “looks right.” A pilot’s reject log should cite the specific reject rule each rejected clip failed, which lets a buyer confirm the reject rule itself is workable rather than either too loose or too strict for the task.

A pretraining corpus is checked for diversity along the same axes whether it is a pilot or full volume: physical scenes, task variety, object categories, and the number and variety of contributors performing them. A pilot’s coverage report should be read against the spec’s condition cells — not as a single aggregate hour count — because a pilot that fills quickly in one cell and stalls in another is telling you exactly where production volume will be slow or expensive to reach.

A pilot cannot prove diversity at full-corpus scale in a small batch — that is a structural limit of evaluating any small sample. What a pilot can show is whether the condition cells in the spec are being filled as designed, and whether the fill rate per cell is fast enough, or slow enough, to tell you something about staffing timelines before you commit to the full volume.

Label quality in a pilot is checked the same way it is checked at full volume: against the acceptance test written into the spec, not against a general sense of whether labels look plausible. If the spec calls for action or hand-pose labels, the pilot should ship labels reviewed against that same acceptance test, so a buyer can confirm the labeling process — not just the capture process — will hold at scale.

Calibration and cross-stream sync are checked on the delivered footage itself, not assumed from a spec sheet. A pilot that states a sync tolerance but does not let a buyer verify it on real delivered clips has not actually proven sync quality. The same applies to camera calibration where multiple views are captured: the pilot should make it possible to confirm calibration held across the sample, not just in a single demo clip.

Failure case coverage is part of the same check. A pretraining corpus that only contains successful attempts is missing a signal models need — what a failed or partial attempt looks like. A pilot should show whether failure cases are captured and labeled as failures, rather than filtered out before delivery.

Every clip in a pilot should carry a traceable consent record — this is a check that applies to every clip in a sample, not a spot check on a few. A pilot is also the right place to confirm anonymization or de-identification steps, where the spec calls for them, are actually applied to delivered footage rather than described only in a policy document.

Format fit is a practical check that is easy to skip and expensive to discover late: does the pilot’s delivery format and schema load cleanly in the training pipeline the buyer actually uses? A pilot delivered in RLDS, LeRobot, WebDataset, or HDF5 should be loaded and spot-checked in the buyer’s own loader before full volume is committed, not assumed compatible from a format name alone.

A pilot evaluates process and coverage — whether the spec, accept criteria, and capture pipeline hold on real footage — rather than directly predicting a trained policy’s real-robot success rate. Whether pretraining data of a given kind improves downstream robot performance at all is a separate, dataset-specific research question that is evaluated at the corpus level, not inferred from a small pilot sample.

NVIDIA’s EgoScale paper is one documented example of researchers studying that link directly: the paper examines how egocentric pretraining data quality and scale relate to downstream manipulation performance, reporting results from training on egocentric human video at scale (EgoScale paper, EgoScale project page). That kind of evaluation happens at full corpus scale, after pretraining, and is not something a buyer should expect a small pilot sample to reproduce on its own.

What a pilot can responsibly tell a buyer is narrower and more immediately useful: whether the specific spec they wrote will produce footage that clears their own accept criteria, at a reject rate and coverage fill they can live with at scale. That is a process question a small, fast batch is well suited to answer — unlike a downstream-performance question, which requires full-scale data and a trained model to evaluate.

After a pilot is evaluated against the checks above, the outcome should map to one of three decisions: go to full volume on the spec as written, adjust the spec and run a second pilot, or stop before committing further budget. Which decision applies depends on where the pilot results landed relative to the written accept criteria — not on a subjective sense of whether the sample “looked good.”

Pilot findings that trigger an adjust decision are read back into the existing skill spec rather than producing a new document. A condition cell with a high reject rate may need a tighter reject rule or clearer capture instructions; a cell that fills slowly may need a longer staffing timeline rather than a shorter one. The signed spec stays the single reference document that recruitment, capture briefs, QA, and delivery are all measured against through every later stage.

A custom pretraining pilot starts the same way a full custom collection does: with a written skill spec naming the task, population, geometry, condition cells, and reject rule. Firsthand runs this as a Spec Pilot — a small, fast first batch captured against that spec, billed per validated hour, and delivered with the same reject log and coverage reporting a full collection gets, so the results can be scored on the same criteria before scaling.

No standing program or long onboarding is required to start a pilot. A focused brief is a normal starting point, and the pilot can be scoped specifically to answer the open questions a buyer has — a rare embodiment, an unfamiliar condition mix, or a task spec they are not yet confident is complete — rather than running a generic sample.

Because the pilot and the full collection share one spec, one accept criteria set, and one billing unit, a buyer scaling from a passed pilot is not taking on new process risk at full volume. The pipeline that cleared the pilot is the same pipeline that runs at scale.