Pretraining vs Mid-Training vs Post-Training Data for Robots

Robot foundation models are typically trained in three stages, and the distinction matters because each stage draws on a different kind of data, at a different scale, for a different purpose. Pretraining builds broad, general priors from the largest and least robot-specific data available. Mid-training narrows those priors to a target embodiment — a specific robot, camera rig and action space — using a comparatively small, aligned dataset. Post-training, often called fine-tuning, specializes the model on the exact tasks, environments and success criteria it will face once deployed.

The practical reason to separate these stages is cost and availability. General priors can be learned from data that does not need to come from the target robot at all, which is why pretraining can use orders of magnitude more data than mid-training or post-training typically can. Robot-specific data is expensive and slow to collect, so the training recipe is designed to need as little of it as possible, spent where it matters most: aligning and specializing, not teaching from scratch.

The three stages do not just differ in data volume — they place different weight on the same underlying capture properties.

• Scale matters most at pretraining, where the data does not need to come from the target robot and can be collected far more broadly than teleoperated robot data allows.
• Diversity across environments, objects, tasks and demonstrators matters most at pretraining, so the model does not overfit to one setting before it ever sees the target robot.
• Calibration and cross-stream synchronization matter at every stage, but become more precise and more embodiment-specific moving from pretraining through mid-training to post-training, where camera position and geometry must match the robot’s actual sensor rig.
• Action labels — hand pose, contact events and task segmentation — matter throughout, but shift from derived labels (as EgoScale used off-the-shelf SLAM and hand-pose estimation on human video) toward precise, task-specific action and success annotation as training moves toward post-training.

NVIDIA’s GEAR lab published EgoScale as a direct study of exactly this question. The recipe pretrains on 20,854 hours of action-labeled egocentric human video — with action labels derived from off-the-shelf SLAM and hand-pose estimation rather than manual annotation — and then follows with a lightweight, aligned mid-training stage on a much smaller amount of real robot data. EgoScale also reports a log-linear scaling law between the amount of human pretraining data and validation loss, correlating with real-robot downstream performance, and a 54% average success improvement over skipping pretraining entirely on a 22-degree-of-freedom dexterous hand.

The "lightweight" framing of mid-training is the key structural fact: EgoScale’s result comes from pairing a very large pretraining layer with a comparatively small, aligned mid-training layer — not from scaling mid-training data itself.

NVIDIA’s GR00T N1 paper describes training on a mixture of data types arranged like a pyramid by abundance and specificity: web-scale data and human video at the wide base, synthetic data in the middle, and real robot data at the narrow top. Mapped to the three training stages, the pyramid’s base corresponds to pretraining, the middle synthetic layer can support either pretraining or mid-training depending on how it is used, and the top real-robot layer corresponds to mid-training and post-training, where alignment to the specific deployed embodiment matters most.

Post-training, often called fine-tuning, is the stage that specializes a pretrained and mid-trained model for its exact deployed job: the specific tasks, environments and success criteria it will actually be evaluated against. Unlike pretraining, which benefits from breadth, post-training data is deliberately narrow and scoped to the target task distribution — more of it does not help if it is not representative of what the deployed model needs to do.

There is no single published volume figure for "enough" post-training data, because the right amount depends entirely on how many distinct tasks, environments and edge cases the deployed model must cover, and how much of that coverage was already established during pretraining and mid-training. This is why post-training data collection is typically scoped through a small pilot batch validated against the exact deployment spec, rather than a fixed target set in advance.

A single custom collection engagement can be scoped to serve more than one stage, provided the spec accounts for the differences up front. Broad, diverse egocentric capture across many environments and tasks serves pretraining. The same collection program, narrowed to a specific camera rig, task framing and environment set matched to a target robot, serves mid-training. Narrowing further to the exact deployed task distribution, with precise action and success annotation, serves post-training.

What does not change across stages is the underlying capture stack: calibrated cameras, cross-stream synchronization, and 3D hand pose with contact events are useful at every stage — what changes is how broad or how narrowly scoped the task and environment distribution needs to be.

Planning starts with naming the stage the dataset is meant to serve, since that determines scope, volume expectations and how narrowly the task and environment distribution should be defined.

• For pretraining: define breadth — the range of environments, objects, tasks and demonstrators needed — rather than a specific robot embodiment.
• For mid-training: define the target robot’s camera position, calibration and action space, and scope the task set to what needs aligning, not a full task distribution.
• For post-training: define the exact deployed task distribution, environments and success criteria the model will be evaluated against.

Every stage runs through the same Spec Pilot process: define the spec, validate it against a small pilot batch, then capture at scale and deliver in the buyer’s chosen format under a buyer-owned perpetual license, billed per validated hour. Starting with pretraining data collection lays out this process in full for the pretraining layer specifically.

The clearest planning mistake is treating all three stages as one undifferentiated "robot data" request. A spec written for pretraining breadth — many environments, many demonstrators, loosely constrained tasks — will not produce a dataset that aligns cleanly to a specific robot’s camera rig, because embodiment alignment was never part of the brief. Conversely, a spec written narrowly around one deployed task will not give a model the broad priors pretraining is meant to establish. Naming the stage first, then writing the spec to match its scale and specificity requirements, is what keeps a custom collection engagement targeted at the problem it is actually meant to solve.

Delivery format planning follows the same stage-first logic. A pretraining dataset destined for large-scale training pipelines is typically delivered as RLDS or WebDataset shards for efficient streaming at scale, while a smaller mid-training or post-training set aligned to one robot may be just as usable in LeRobot format or HDF5, depending on the training stack already in place. Deciding the target training stage before the delivery format avoids re-exporting the same capture twice.