Pretraining Data for Generalist Robot Policies

A generalist robot policy is a single model trained to perform many tasks, often across more than one robot embodiment, rather than a narrow policy fine-tuned for one task on one specific robot. Instead of learning "pick up this part on this arm," a generalist policy aims to learn transferable priors — how objects behave under contact, how a task decomposes into steps, how a viewpoint maps to an action — that carry across tasks and, ideally, across robot hardware.

NVIDIA's GR00T N1 paper frames this directly: it describes training a generalist humanoid robot foundation model on a mixture of data types, rather than a single robot's demonstration set, precisely because a generalist policy needs to have seen more variation than any single-robot, single-task dataset can supply.

The word "generalist" is doing real work here, and it is worth separating from "multi-task." A policy can be multi-task on a single robot (many skills, one embodiment) without being a generalist in the cross-embodiment sense; a true generalist policy is expected to transfer some of what it learned on one robot, or from human video, to a different embodiment with only a small amount of embodiment-specific alignment data. That cross-embodiment transfer is the harder, more data-hungry version of the problem, and it is the one this guide focuses on.

Because a generalist policy has to generalize across tasks and, often, embodiments, its pretraining stage needs breadth before it needs precision. This is the opposite emphasis from post-training, which narrows to the exact tasks and success criteria a deployed policy must handle. At pretraining time, the priority is coverage: many scenes, many objects, many tasks, many people, and — where the goal is cross-embodiment transfer — signal that is not tied to one robot's action space.

Egocentric human video fits this requirement well because it can be collected at a scale and diversity that teleoperated robot data typically cannot reach, and because it is embodiment-agnostic at the point of collection: a person's hands performing a task do not commit the resulting footage to any one robot's gripper geometry or degrees of freedom. That is what makes it useful as a shared pretraining layer across multiple downstream embodiments, rather than data that only benefits the one robot that collected it.

The embodiment gap is the mismatch between a human body — hands, arms, sensing — and a robot's hardware, such as a parallel-jaw gripper, different arm kinematics, or a different camera viewpoint. Human demonstration data, including egocentric human video, carries this gap because it was recorded from a human's body rather than the target robot's, so a retargeting or alignment step is needed before that data's action information matches what any specific robot can execute.

For a generalist policy, the embodiment gap shows up twice: once between human pretraining data and any robot at all, and again between one robot embodiment and another if the policy is meant to transfer across hardware. This is why cross-embodiment pretraining data is not simply "more data" �� it is data structured so that shared priors (how a task unfolds, what contact looks like) can be learned once and then aligned separately to each target embodiment, rather than relearned from scratch per robot.

EgoScale's published result is the clearest public evidence that this bridging is solvable at scale: large-scale human pretraining followed by a small, aligned mid-training stage on real robot data measurably lifted real-robot task success. See the embodiment gap glossary entry for a fuller definition and how it differs from the sim-to-real gap.

Three publicly documented research efforts each mix pretraining sources differently, and comparing them is useful for understanding the design space a generalist pretraining plan sits inside.

The shapes differ, but the underlying logic does not: each pairs a large, broad, human-or-web-scale layer with a much smaller, embodiment-specific layer for alignment. GR00T N1's pyramid and EgoScale's two-stage recipe are describing the same wide-base-to-narrow-top structure from two different angles — one as a static data mixture, the other as a training sequence.

GEN-0's own framing is explicit on this point: data quality and diversity matter more than sheer volume. For a generalist policy specifically, this matters more than it would for a narrow, single-task policy, because the entire premise of "generalist" is that the model will encounter scenes, objects and task variations it did not see verbatim during training. A large but narrow pretraining corpus — many hours of a handful of repeated tasks in a handful of environments — teaches depth on those tasks but does little to prepare a policy for the breadth a generalist deployment will actually face.

EgoScale's reported log-linear scaling law is often read as "more hours is always better," but the more precise reading is that more hours of diverse, action-labeled egocentric video kept producing measurable gains — the diversity and labeling were present throughout the scaling curve the paper reports, not a separate variable held constant while only volume increased. A generalist pretraining plan should treat diversity across scenes, objects, tasks and embodiments as a primary design axis, not an afterthought once an hour target is chosen.

For egocentric human video to function as generalist pretraining data — rather than just raw footage — several properties matter alongside coverage breadth.

• Diversity across environments, objects, tasks and demonstrators, so the model does not overfit to one kitchen, one grasp style or one body type.
• Action labels — hand pose, contact events and task segmentation — so the video carries structure a policy can learn from, not just pixels. GR00T N1's latent-action and inverse-dynamics techniques let unlabeled video contribute some signal, but direct annotation gives a more explicit supervisory signal than pseudo-labels inferred after the fact.
• Camera calibration, so intrinsics and extrinsics are known and downstream depth or pose estimation from the footage is reliable across every embodiment the data might eventually support.
• Cross-stream synchronization, so multiple camera or sensor streams captured together stay aligned in time and can be fused rather than treated as loosely related recordings.

Because a generalist policy is meant to serve more than one downstream task or embodiment, these properties matter even more than they do for a single-robot pretraining plan: a labeling or calibration gap that a narrow policy might absorb becomes a compounding problem once the same base layer is expected to support several different alignment stages later.

Aligned mid-training capture is the bridge step between broad, cross-embodiment pretraining data and a specific target robot. Rather than general egocentric footage across many tasks and environments, mid-training capture narrows in: camera position and calibration matched to the robot's eventual sensor rig, task framing that mirrors what the deployed robot will actually attempt, and enough coverage to align the general priors learned during pretraining to that specific setup.

This is the layer that does most of the work of closing the embodiment gap for a generalist policy. EgoScale's recipe describes this stage as "lightweight" precisely because it does not need to teach the model everything from scratch — it only needs to teach the model how its already-broad priors map onto one embodiment's actuators, kinematics and camera geometry. For a policy meant to serve multiple embodiments, this implies running a separate, comparatively small aligned mid-training pass per target robot, on top of one shared pretraining base.

Public egocentric corpora are a legitimate starting point for the base pretraining layer, and useful as a reference for what scale has looked like in published research. But they are fixed: the tasks, environments, objects and camera rigs were chosen by whoever collected them, not by the team building a specific generalist policy.

A generalist pretraining plan that leans entirely on public corpora inherits whatever coverage gaps those corpora happen to have �� which embodiments they were framed for, which task types are overrepresented, which demographic or environmental variation is missing. Custom pretraining collection exists for teams that need coverage a public corpus does not have: a specific task distribution, a specific set of environments, or camera and calibration choices matched to more than one downstream embodiment at once. The typical path is a small pilot batch scoped against a written spec — target scenes, tasks, objects and people — reviewed for calibration, synchronization and label quality before scaling into the full base layer.