How Does Pretraining Data Affect Robot Generalization?

Generalization is how well a trained robot policy performs on scenes, objects or tasks that differ from what it saw during training. In-distribution performance describes a policy executing a task under conditions close to its training data — the same type of room, similar objects, similar lighting. Out-of-distribution performance describes the same policy facing a new room, an unfamiliar object shape, or a task variation it was never directly shown.

A policy can score well on in-distribution evaluation while failing badly out of distribution, because strong performance on familiar conditions does not by itself prove the model learned the underlying task rather than memorizing the narrow slice of conditions it was trained on. That gap — between performance inside and outside the training distribution — is the practical thing pretraining data composition is trying to close.

Pretraining data shapes generalization by setting the boundaries of what a policy has evidence for. A policy only generalizes to conditions, objects and task variations that are represented somewhere in its pretraining data, even indirectly — it cannot transfer to something it has no evidence of, no matter how much total data it saw. This is why dataset diversity matters in robot learning: a large but narrow corpus (one room, one lighting setup, a small set of objects) teaches a policy that narrow slice, regardless of its hour count.

Four properties of pretraining data composition matter most for how a policy generalizes once trained: how much data exists (scale), how varied that data is across the axes a deployed robot will actually encounter (diversity), how accurate and usable the action labels attached to that data are (label quality), and whether the data includes failed attempts and recoveries alongside clean successes (failure coverage). Each is covered below with what published research and this site’s own capture methodology say about it.

Within specific published studies, yes — but the evidence points to scale interacting with composition rather than acting alone. NVIDIA’s EgoScale, pretrained on 20,854 hours of action-labeled egocentric human video with labels derived from off-the-shelf SLAM and hand-pose estimation, reports a log-linear scaling law: as the amount of pretraining video increases log-linearly, validation loss keeps falling, and the paper states this trend correlates with real-robot downstream task performance rather than being purely an offline metric. EgoScale also reports a 54% average success improvement over skipping pretraining entirely, measured on a 22-degree-of-freedom dexterous hand.

Generalist AI’s GEN-0, described in the company’s own November 2025 announcement as trained on more than 270,000 hours of real-world manipulation data and growing by over 10,000 hours per week, adds an important qualifier directly from its own materials: data quality and diversity matter more than sheer volume. Read together, these two findings are complementary rather than contradictory — scale helps within a given data distribution, but the composition of that data determines how much a given amount of scale is actually worth for generalization.

Every EgoScale and GEN-0 fact on this page traces to their own published sources, linked below. No hour count, benchmark, or customer name from either study is attributed to Firsthand, and no numeric result from either paper is restated as something this site has produced or measured.

Diversity is not one property but several independent axes, and a gap in one axis is not compensated by strength in another — a corpus with many scenes but one task type is still narrow, just along a different dimension than a corpus with many tasks filmed in a single room. How to plan diversity in a robot pretraining dataset breaks these axes down for collection planning; the table below focuses specifically on how each axis connects to a concrete generalization failure mode.

Diversity in what was recorded only helps if the labels describing what happened in that recording are accurate enough to teach a usable signal. EgoScale’s pretraining value depends on its action labels — derived from SLAM and hand-pose estimation — being usable priors for the mid-training stage that follows; volume without usable labels would not produce the same scaling result the paper reports. Calibration and synchronization between cameras and sensors matter for the same reason: camera calibration and multi-sensor sync error are what make hand-pose and action labels consistent across episodes, scenes and contributors, rather than drifting in ways that corrupt the signal a policy learns from.

This is also why label quality and diversity are not separable concerns in practice. A dataset can be diverse in scenes and objects while still teaching a brittle policy if action labels are inconsistent across contributors or episodes, because the model is learning from noisy supervision even though the underlying footage spans the right conditions. Consistent labeling across a diverse corpus is what lets the diversity actually translate into transferable priors rather than just varied-looking footage.

A pretraining corpus built entirely from clean successes does not teach a model what a stall, a missed grasp, or a recovery attempt looks like. Why failure cases matter in robot learning covers this directly: when a policy only ever sees successful executions, it has no evidence for what to do when something goes wrong in a new, out-of-distribution setting — which is exactly where unexpected outcomes are more likely. How to capture failure cases in robot training data describes how failure and recovery behavior is defined and captured as a distinct category in a collection spec, rather than left to accumulate by accident.

Failure-case coverage behaves like a diversity axis in practice: as collection volume grows without deliberate planning, the share of a corpus made up of clean successes tends to rise on its own, because failures are rarer and easier to skip than to seek out. A spec that defines what counts as a failure case and sets a minimum coverage target for it is what keeps this axis from being crowded out as a program scales.

Several independent research efforts now study how human-video and large-scale manipulation data affect robot policy performance, and treating them together is useful because no single study is the final word. NVIDIA EgoScale and HumanScale (arXiv 2606.20521) are two distinct, separately published research lines studying the scaling behavior of human-video pretraining for robot policies — together they are evidence that this question is an active area of robotics research, not a single lab’s isolated result. GEN-0’s own announcement frames its 270,000+ hour dataset around the claim that quality and diversity matter more than sheer volume. GR00T N1 (arXiv 2503.14734) describes a data pyramid that separates web data and human video, synthetic data, and real robot data into distinct layers by abundance and specificity, implying that the mix of data types — not just total hours — shapes what a model learns and how far it generalizes.

Each figure above is attributed to its own official source, linked in Sources below. No benchmark, hour count, or result from any of these studies is presented as a Firsthand result, and Firsthand does not claim to have produced any of the datasets named here.

Each of the mechanisms above — diversity, label quality, failure coverage — is a named property of a skill spec and a QA step in a custom collection, not an implicit outcome of filming more hours.

Planning a pretraining collection for generalization starts from the same Spec Pilot process used for any custom collection on this site: define the target environments, tasks, objects and contributor mix in a skill spec, validate the approach and label quality against a small pilot batch, then capture at scale and deliver in the buyer’s chosen format — RLDS, LeRobot, WebDataset or HDF5 — under a buyer-owned perpetual license, billed per validated hour.

Given what the published evidence above actually supports, the spec is better built around closing specific coverage gaps than around a raw hour target: naming which scenes, objects, task variations and contributor types a model is weakest on, then setting minimum coverage for each, including a defined share for failure and recovery cases. Pretraining data collection and how to plan diversity in a robot pretraining dataset cover how that spec-first approach is structured for a custom, on-demand collection.