Why Physical AI Pilots Stall on Training Data

A physical AI program usually moves through three phases: find where a robot can create value, prove it in a real setting, then roll it out. The middle phase — the pilot — is where most programs slow down. The model is often strong in general, but the pilot asks it to do one customer’s tasks, on that customer’s objects, in that customer’s rooms, and the data that describes those exact conditions usually does not exist yet.

That gap is the pilot data bottleneck. A foundation model pretrained on broad data carries useful priors about how hands, objects and tasks behave, but the last stretch of reliability comes from task- and environment-specific examples. Those examples have to be captured, labeled, checked and delivered before the pilot can show the result the customer is waiting for. When that data takes months, the pilot waits months.

Pretraining data is built for breadth: many tasks, many scenes, many people. That breadth is what gives a robot foundation model its general priors, and it is covered in depth in our guide to robot pretraining data. A pilot needs the opposite property — depth on a narrow slice. The customer does not care that the model has seen thousands of kitchens; it cares whether the robot can do the one packing, sorting or assembly step in this building.

Three things tend to be missing from general data at pilot time. The first is the exact object set: the specific parts, packaging or tools the site uses. The second is the exact environment: lighting, clutter, fixtures and reach distances that shape how the task is actually performed. The third is failure coverage: the fumbles, regrasps and recoveries that happen on that task, which general datasets rarely capture on purpose. Our guide on why failure cases matter explains why a policy that has only seen clean successes tends to break at the first unusual moment.

The data a pilot needs is defined by the pilot’s tasks, not by a general catalog. In practice it is task-specific egocentric human demonstration data, captured in conditions that match the deployment, with labels the training pipeline can consume directly.

Egocentric capture is a good fit for pilot data because people can perform the target task today, in a matched environment, without the robot. Our standard rig pairs a head-mounted global-shutter 4K camera with wrist cameras, stereo, depth and IMU streams, synchronized over PTP to a 1.12 ms median and 1.94 ms p99 residual, with a 2.0 ms reject ceiling. Deliveries carry 3D hand pose and action-segment labels so the data plugs into imitation-learning and post-training pipelines.

Fast custom collection shortens the loop between noticing a gap in the pilot and getting data that closes it. The loop starts with a skill spec: a written description of the task, the environment, the objects, the capture setup and the accept/reject rule. Once a spec is signed, first validated data arrives in a median of 48 hours where contributor coverage is deep, and in 3–4 weeks where coverage is thin and contributors need to be recruited first.

• Write the gap down as a skill spec, naming the task by verb and object rather than by category.
• Run a small spec pilot against that spec to confirm the capture and labels are what the training team needs.
• Scale the collection against the same spec once the pilot data trains cleanly.
• Feed the validated batches into the pilot’s training loop and add new spec cells as new failures appear.

The key is that speed does not come from skipping steps. Every batch still passes QA, consent verification and anonymization before delivery. What makes it fast is that the capture network, the pipeline and the process already exist, so a new spec starts from a running system rather than a blank page.

A pilot that surfaces failures is doing its job. Each observed failure is a precise description of data the model is missing: this object, this grasp, this recovery. Turning those observations into new spec cells, capturing against them, and retraining is the flywheel that moves a pilot toward deployment. The faster each turn of that loop, the fewer weeks the pilot spends stuck on one failure mode.

Because every Firsthand collection is defined by a skill spec and delivered with a data card, each turn of the loop is documented: which cells were captured, how many validated hours passed QA, and what was rejected and why. That record matters later, when the pilot becomes a deployment and the team needs to explain what the production model was trained on.

None of these items has to be perfect before the first conversation. A skill spec is written collaboratively, and the spec pilot exists precisely to find the parts of the spec that are wrong before the collection scales.

Pilot data is meant to end up in a production model. If it arrives under a research-only license, the team has to recollect or renegotiate before deployment, which reintroduces the delay the pilot was trying to avoid. Firsthand delivers under a perpetual, buyer-owned license with a consent chain for every contributor, so pilot data can move straight into the model that ships.