How to Scale a Robotics Data Collection From Pilot to Production

A robotics data pilot — a spec pilot — is a small, fast first batch of footage captured against a written skill spec, used to prove the spec and its accept criteria before committing to full volume. It surfaces the real reject rate, coverage fill, and sync numbers on real footage at low volume, rather than asking a buyer to commit to scale on a pitch deck.

A pilot is not a free sample of footage that already exists. It is captured against the exact same skill spec — the same task, geometry, condition cells, reject rule, and acceptance test — that a full custom collection would use, which is what makes its results a legitimate preview of what production volume will actually look like.

See pilot data collection for the full definition and a side-by-side comparison against a full custom collection.

Committing to full volume on a spec that has never been executed is a bet on a pitch deck, not on data. A pilot turns that bet into a measured result — the exact reject rate, sync residual, and coverage fill a buyer would get at scale — while the cost of being wrong is still small.

A pilot is optional, not mandatory. No standing program is required to start a custom collection, and a focused brief is a normal starting point. A buyer who has run a similar collection before, or whose spec covers a well-served population and a familiar task, often moves straight to full volume. A buyer testing an unfamiliar condition mix, a rare embodiment, or a spec they are not yet confident is complete gets more value from seeing real reject and fill numbers on a small batch before committing the full budget.

The actual value of running a pilot first is not the sample footage itself — it is the process guarantee that comes with it. A buyer scaling from a pilot to full volume is not taking on new process risk, because the process at scale is provably the same process that already passed at small volume. The spec, the accept criteria, and the billing unit all carry over unchanged.

A pilot exists to answer a short list of concrete questions before a buyer commits to full-volume budget: does the delivered footage clear the written accept criteria, what is the real reject rate against the reject rule, how quickly does each condition cell in the spec actually fill, and does the sync and calibration hold at the measured tolerance rather than only in a demo reel.

Those questions map directly onto the reporting a pilot ships with: a reject log with reasons rather than just pass counts, and coverage reported against the spec’s condition cells rather than as a single aggregate hour count. A pilot that clears its accept criteria on paper but cannot show that reporting has not actually proven anything, because there is no way to check whether the result would hold at higher volume.

A skill spec is the written contract the whole collection is measured against: it names the exact task by verb and noun, states the geometry that must stay in frame, budgets hours per condition cell, and writes the reject rule and one acceptance test before capture starts. A pilot is captured against that same document, which means pilot findings do not require a new spec — they are read back into the existing one.

In practice, that means a pilot rarely rewrites a spec from scratch. It confirms the parts that were right, and it flags the parts that need a decision: a condition cell whose reject rate ran high enough that the reject rule may need tightening or the capture instructions may need clarifying, or a cell that filled slower than expected and needs a longer staffing timeline rather than a shorter one.

A signed spec functions as the single reference document for every later stage — recruitment screens contributors against its named population, capture briefs contributors against its geometry and reject rule, QA measures delivered items against its acceptance test, and delivery packages the data according to its named format and schema. Carrying pilot findings back into that same document, rather than into a separate report, is what keeps the production run measured against the exact thing the pilot proved.

Scaling volume up does not mean scaling scrutiny down. Every batch, large or small, is measured against the same signed accept and reject criteria from the spec stage, and every batch produces its own reject log. A high-volume program does not trade thoroughness for throughput; it runs the same per-batch check at every batch, however many batches the full volume requires.

Billing follows the same discipline at every scale: the unit is a validated hour, meaning footage that has already cleared the spec’s accept criteria, not a raw hour of capture before anyone has checked it. A buyer scaling from a pilot to a full program is paying for the same unit at both stages, which is part of what keeps the two comparable.

This is also why a spec is written by condition cell rather than as one aggregate hour count. A policy fails on the conditions it never saw, not on the ones it saw a lot of, so budgeting hours per cell — daylight, low light, cluttered, failure cases — surfaces exactly where coverage is thin before training ever does. Fill rates are reported against that same document at every batch, so both the buyer and the pipeline are looking at the same coverage picture as volume scales up.

That per-batch reporting is what turns scale from a black box into something a buyer can steer mid-collection. If a batch report shows the low-light cell filling slower than daylight, that is visible before the whole volume is spent chasing an easy cell — the buyer and the pipeline can decide together whether to extend the timeline for the thin cell or accept a documented gap, rather than discovering the gap only once the finished dataset is already in hand.

Every sensor shares one PTP (IEEE 1588) clock domain with hardware-triggered exposures, so RGB, depth, IMU, and pose all carry timestamps on the same timebase. The measured residual offset between streams runs a median of 1.12 ms and a p99 of 1.94 ms, against a 2.0 ms reject ceiling — and that ceiling is enforced per episode, not audited after the whole batch is delivered, at every stage from pilot through full production volume.

Calibration follows the same discipline as sync: it is checked and logged per session rather than assumed to hold from a one-time setup. Sensor rigs drift with heat, handling, and time, so a rig calibrated correctly on day one of a production run is not guaranteed to still be accurate on day thirty — which is why calibration state, like sync residual, is recorded against every episode rather than treated as a fixed property of the hardware.

Recruitment and staffing draw on a screened contributor pool matched to every cell of the spec — language, accent, age band, device, environment, expertise — and thin or under-served segments get a realistic staffing timeline rather than an optimistic one, agreed up front during specification and confirmed by the pilot before production volume is committed.

A diversity plan also has to hold up across an entire production run, not just the pilot sample. A vendor that front-loads its most convenient scenes and demonstrators into the pilot batch, then fills the remainder of a production order with whatever is fastest to capture, quietly narrows the diversity that made the pilot look strong in the first place. Checking delivered footage batch by batch against the same spec’s condition-cell budget, rather than only at the end of the order, is what catches that drift before it reaches a training run.

Scale also means drawing from a genuinely global pool rather than a single regional labor market, which is what makes rare condition cells staffable at production volume at all. A spec that needs a specific language, accent, home layout, or device is not searching a local pool for a coincidental match — it is matching against a pool that spans many countries by design, so a thin cell in one region can still be filled from contributors elsewhere who meet the spec’s criteria.

Every contributor signs a release before capture, and the same standard covers anyone who appears in frame incidentally — family members, coworkers, bystanders — not only the person performing the task under contract. That consent chain does not change shape between a pilot and a production run; it is applied per contributor and per session at every volume.

Faces, license plates, and screens are detected, tracked, and irreversibly blurred, with a human reviewer confirming the result rather than trusting automated detection alone, at every batch of a scaling collection. The output at each stage is a validated set plus a reject log and an anonymization record a buyer can audit, alongside the signed consent releases that make up the provenance behind the dataset.

Consent and anonymization are checked as two separate, sequential steps rather than one combined pass, because they answer different questions — whether a person agreed to be recorded, and whether anyone who did not consent is still identifiable in the delivered footage. Running both checks independently at every batch, rather than assuming a pilot’s clean result guarantees a production batch’s, is what keeps the consent chain intact as volume grows.

Delivery ships in the format specified during the spec stage — RLDS, LeRobot, WebDataset, HDF5, zarr, Rerun, WAV, JSONL, and others — with the per-item metadata, the signed consent releases, the anonymization record, and a data card recording jurisdictions and provenance, all under a perpetual, buyer-owned commercial license rather than a shared or revocable one.

A production run typically ships as a rolling series of batches rather than one delivery at the end, with a reject log shipped alongside every batch. That per-batch delivery is what lets a buyer start integrating data into a training pipeline while the rest of the volume is still being captured, instead of waiting for the entire production run to finish before anything is usable.

Before committing a pilot’s findings to full production volume, the following are worth checking against the pilot’s own results rather than against the original spec document alone.

The most common mistake is re-negotiating the spec between the pilot and the production run instead of carrying the same document forward. If the accept criteria, condition-cell budget, or reject rule change between the two, the pilot no longer proves anything about the production batch — it proved a different collection.

A second common mistake is reading a pilot’s aggregate result without checking coverage cell by cell. A pilot that clears its accept criteria overall can still have filled easy cells quickly and thin cells barely at all — a result that only shows up if fill rate is reported per condition cell rather than as one combined number.

A third is treating the pilot’s sample size as representative of production sync and calibration performance without continuing to measure both at scale. A small pilot batch, captured over a short window, may not surface the calibration drift that shows up only after many sessions and many days of capture — which is why sync residual and calibration state are logged per episode throughout a production run, not just during the pilot.

A fourth is assuming that because a pilot passed, staffing timelines for thin condition cells will hold at full volume without being revisited. A pilot batch small enough to staff quickly from a thin population does not prove that the same population can be staffed at the multiple needed for full production volume — that staffing timeline is a separate question, agreed during specification, that a pilot’s speed does not automatically answer.

Where coverage is already deep, the median time from a signed spec to a first validated batch is 48 hours, because contributors, rigs, and reviewers are already in place before a spec lands. Thin or under-served environments — such as construction, clinical, agriculture, or hospitality — typically take three to four weeks, because vetting operators there is the actual bottleneck, not the pipeline itself.

Moving from a cleared pilot into production does not restart that clock. The same contributors, rigs, and reviewers that ran the pilot continue into the production run, so scaling volume is a matter of staffing more sessions against the same spec rather than re-establishing the pipeline from zero.

See fast, on-demand data collection for the full breakdown of what drives the 48-hour figure and where thin coverage changes the timeline.