Fine-Tuning Data for Robot Policies

Pretraining is broad and human-sourced: a large corpus built for coverage and scale across many tasks, scenes, and often embodiments, with no single deployment in mind. Mid-training is a smaller, tightly aligned stage pairing matched human and robot capture to one target embodiment — EgoScale’s published recipe illustrates the ratio, roughly 50 hours of human video plus about 4 hours of robot data across 344 tabletop tasks, layered on top of its 20,854-hour pretraining corpus.

Fine-tuning (post-training) narrows further still. Where pretraining optimizes for breadth and mid-training optimizes for embodiment alignment, fine-tuning optimizes for reliable performance on one specific deployment task, robot, and environment. It is the smallest and most specific stage, and the one judged on how precisely it represents the target task rather than on how much ground it covers. See pretraining vs mid-training vs post-training data for robots for the full three-stage comparison.

Public pretraining corpora are built for breadth, not for any one deployment’s failure modes. A policy that is underperforming on a specific task, object, or environment needs data captured against that exact gap, which is by definition not what a general-purpose public corpus was built to cover, and several of the largest public egocentric corpora ship under research-only terms in the first place.

A custom collection starts from the opposite direction: a written skill spec names the task, the robot, the environment, and the edge cases a program is actually failing on, and capture is scoped tightly around that spec rather than drawn from whatever footage happens to already exist. That is the difference between fine-tuning data and pretraining data in practice — precision over volume.

A fine-tuning spec names the target task and its success criteria precisely, since post-training narrows to one deployment rather than general coverage. It specifies the objects, layouts, and environment variation the deployment task will actually encounter — fine-tuning does not rely on pretraining-scale breadth to fill gaps, so this variation has to be captured directly.

It also specifies the target robot embodiment the data should be captured on or matched to, and — distinct from pretraining or mid-training specs — it names the failure modes and recovery behaviors the policy currently struggles with, so the batch is built around the exact weaknesses a deployed or evaluated model is showing, not just a clean demonstration of the task done correctly.

Every batch runs against machine-checked reject criteria before a human reviewer checks each episode against the named skill spec, and rejected episodes are recaptured rather than shipped with a caveat.

The same signal quality applies at every training stage, including fine-tuning: synchronized streams hardware-aligned under a 2.0 millisecond sync ceiling, with 3D hand pose, contact events, and action segments captured live on the rig rather than estimated after the fact.

3D hand pose is tracked per hand against synchronized depth, with a per-joint visibility flag so occluded fingers are marked rather than guessed. Contact and grasp events mark the moment of hand-object interaction, and action segments break each task into start- and end-timestamped spans rather than one tag covering the whole clip. What changes at the fine-tuning stage is scope — how narrow the task and environment coverage is — not label fidelity or stream count.

A Spec Pilot is a small, fast first batch that proves the spec and accept criteria hold up on real footage before a fine-tuning order scales to full volume. Because fine-tuning batches are already small and precisely scoped, a Spec Pilot miss is cheap to catch early and expensive to discover only after a full batch ships against the wrong definition of the task.

Most fine-tuning orders start this way: a written spec for the target task and its edge cases, a Spec Pilot that validates coverage and the reject rate on real footage with a real timeline figure, and then a scaled order once the spec is proven.

Fine-tuning data ships in RLDS, LeRobot format, WebDataset, or HDF5 — the same delivery formats used for pretraining and mid-training data from the same capture pipeline, so a program can move between training stages without switching providers or re-converting formats.

A fine-tuning batch ships under a perpetual, buyer-owned commercial license — the same licensing terms used across pretraining and mid-training data from this pipeline, so a program does not need to track different rights per training stage.

Provenance and the documented consent chain behind each episode carry through to fine-tuning deliveries the same way they do for any custom egocentric collection.

Start with a written spec naming the target task, robot embodiment, environment, and the specific failure or recovery cases the batch needs to cover. Most engagements begin with a Spec Pilot — a small, fast first batch that proves the spec and accept criteria hold up on real footage — before scaling to the full fine-tuning order.