How to Build a Robot Pretraining Data Mixture

A pretraining data mixture is the combination of data sources a foundation-model team blends into a single pretraining run: typically large-scale egocentric human video, some amount of synthetic or simulated robot data, and a smaller slice of real robot teleoperation data. No single source, on its own, currently supplies both the scale and the embodiment specificity a robot foundation model needs, which is why published recipes combine several sources rather than pretraining on one.

The mixture question is distinct from the single-source sizing question covered elsewhere on this site. Knowing that EgoScale pretrains on 20,854 hours or that GEN-0 draws on 270,000+ hours (arXiv 2602.16710; Generalist AI official blog) does not by itself say how much of a team's own pretraining budget should go to human video versus synthetic versus real robot data. That ratio is what a mixture plan decides.

NVIDIA's GR00T N1 paper (arXiv 2503.14734) describes training its vision-language-action model on a data pyramid: a wide base of large-scale, heterogeneous data (web-scale and egocentric human video) narrowing toward a smaller top layer of real robot trajectories for the target embodiment, with synthetic and simulated robot data forming a middle layer between the two. The shape of the pyramid — wide base, narrow top — reflects that the largest-volume data is also the most general, while the smallest-volume data is the most specific to the exact robot and task.

The pyramid framing matters for planning a mixture because it reframes the sizing question. Instead of asking "how many total hours," a team building against a pyramid asks how much of each layer it needs, since each layer is doing a different job: the base layer teaches general visual and manipulation priors, the middle layer adds robot-specific dynamics cheaply through simulation, and the top layer aligns the model to the exact hardware and tasks it will actually run.

Egocentric human video sits at or near the base of the pyramid because it is the most scalable source: it does not require a robot, a teleoperation rig, or a simulator, and one recording session can generate hours of footage across many scenes, tasks and objects. EgoScale's finding that pretraining on egocentric human video plus a small aligned mid-training slice of matched human-and-robot data improved a 22-degree-of-freedom hand task by 54% (arXiv 2602.16710, NVIDIA GEAR) is direct evidence that this layer transfers into robot performance, not just visual pretraining.

Because egocentric human video is the layer easiest to scale and diversify, it is also the layer where a custom collection typically has the most leverage: a team can specify exactly which scenes, tasks, objects and people it needs covered, at first-person viewpoints matched to the robot's planned camera placement, without needing a functioning robot or simulator to be ready first.

Synthetic and simulated robot data fills the pyramid's middle layer: it is cheaper to generate than real robot teleoperation and can cover robot-specific dynamics and edge cases at higher volume than a physical rig allows, but it carries a sim-to-real gap that limits how far it can substitute for real demonstrations. Real robot data — typically collected via teleoperation on the target hardware — sits at the pyramid's narrow top: smallest in volume, but the only layer that captures the exact embodiment, actuator dynamics and sensor characteristics the deployed policy will see.

EgoScale's mid-training stage is a concrete example of how a small real-robot layer gets used: roughly 4 hours of matched robot demonstrations, paired with roughly 50 hours of aligned human video across 344 tabletop tasks, as reported in the paper. That is not the model's primary source of robot-specific learning volume — it is the alignment bridge between the large human-video base and the model's eventual robot deployment.

GEN-0's official blog reports that its scaling to 270,000+ hours was accompanied by an emphasis on data quality and diversity, not hour count alone. EgoScale's scaling-law result is log-linear, meaning early hours in an underrepresented condition move performance more than additional hours in an already well-covered condition — a diversity argument, not a volume argument. Read together, both sources point the same way: a mixture plan should prioritize filling diversity gaps and enforcing quality checks within each layer before adding raw volume to a layer that is already well covered.

The three training stages this site covers elsewhere — pretraining, mid-training and post-training — map onto the pyramid's layers with different mixture ratios at each stage. Pretraining leans heaviest on the wide base (egocentric human video, plus general visual data), mid-training is a deliberately small, tightly aligned slice mixing matched human and robot data for embodiment alignment, and post-training is scoped almost entirely to real robot demonstrations for the specific skills a policy needs.

• Pretraining mixture: mostly egocentric human video, some synthetic/simulated data, little or no real robot data
• Mid-training mixture: a small, matched pair of human video and real robot demonstrations for alignment (EgoScale: ~50h human + ~4h robot across 344 tasks)
• Post-training mixture: predominantly real robot demonstrations for the target skill, per the 50–100 validated hours per skill starting point used elsewhere on this site

Treating these as three separate mixture-planning exercises, rather than one aggregate ratio, keeps each stage's data spend aligned to what that stage is actually training for.

• Treating one published ratio (e.g. GR00T N1's or EgoScale's) as a universal target rather than a reference built for a specific model and task set
• Adding raw hours to the largest layer without checking whether it fills a diversity gap or just duplicates already-covered conditions
• Skipping the mid-training alignment layer and expecting a human-video-pretrained model to transfer directly to a robot without any matched human-and-robot data
• Sizing post-training against the pretraining corpus's scale instead of the specific skill's own evaluation results

The most consequential of these is usually the first: a team that copies GR00T N1's or EgoScale's published composition without adapting it to its own target embodiment and task set ends up over-invested in coverage it does not need and under-invested in coverage it does. Published mixtures are useful as evidence that a layered approach works, not as a template to replicate exactly.

Because the egocentric human-video layer is the most scalable part of the mixture, it is also the layer most amenable to a custom collection scoped to exactly the coverage a mixture plan calls for: specific scenes, tasks, objects, people and camera viewpoints, delivered as calibrated, synchronized, action-labeled footage rather than raw video. Sourcing this layer on demand — rather than relying solely on public corpora built for someone else's model — lets a team fill the diversity gaps a mixture plan identifies instead of settling for whatever public data happens to be available.

A practical starting point is the same pilot-then-scale approach used for sizing any pretraining collection: specify the target mixture composition and coverage cells first, run a small pilot to validate calibration, sync and label quality across that composition, then scale into the base layer once quality is confirmed.

Writing the mixture plan down before commissioning any capture also makes it possible to check coverage against the plan as footage arrives, rather than discovering gaps only after a model has already trained on an incomplete base layer. A spec that names the target scenes, tasks, objects, people and camera viewpoints for the human-video layer gives a collection partner a concrete brief to fill, and gives the team commissioning it a way to confirm — cell by cell — that the delivered footage actually matches the composition the mixture plan called for, rather than a plausible-looking pile of egocentric video that happens to be missing exactly the diversity the top layers of the pyramid depend on.

A mixture plan is not static across a project. The composition that makes sense for an initial pretraining run — heavily weighted toward broad, general-purpose egocentric human video — is deliberately not the composition used once a team moves into aligned mid-training or task-specific post-training. EgoScale's own recipe illustrates this shift: a large, general 20,854-hour egocentric corpus for pretraining is followed by a much smaller, narrowly scoped aligned mid-training slice — roughly 50 hours of human video paired with about 4 hours of matched robot demonstrations across 344 tabletop tasks — before any task-specific fine-tuning happens on top.

Treating this as one continuous mixture plan, rather than three unrelated data-buying decisions, is what makes the later stages efficient. If the pretraining layer was collected with a coverage spec in mind, some of the same viewpoints, environments and object sets can be reused or lightly extended for the aligned mid-training slice, rather than starting mid-training sourcing from a blank page. The mixture composition narrows in scope and grows in specificity at each stage, but the underlying discipline — naming what the data needs to cover before commissioning it — carries through all three.