What Is the GR00T N1 Data Pyramid?

The GR00T N1 data pyramid is the training-data structure NVIDIA describes in its GR00T N1 paper (arXiv 2503.14734) for pretraining a generalist vision-language-action model for humanoid robots. Rather than pretraining on a single data source, the paper organizes training data into three layers stacked by volume and embodiment specificity: web data and human video form the widest base, synthetic data occupies a middle layer, and real robot trajectories sit at the narrow top.

The pyramid shape is the key idea: volume and embodiment specificity move in opposite directions. The base layer is largest in volume but least specific to any one robot, while the top layer is smallest in volume but most specific to the exact hardware, sensors and actuators the deployed policy will run on. Each layer contributes something the others cannot supply as cheaply, which is why the paper combines all three rather than relying on one.

For teams evaluating what to build first, the pyramid is best read as a sourcing checklist rather than a strict pipeline order: identify which layer is currently thinnest for the target embodiment and target task set, then source against that gap specifically, since the base layer is almost always the cheapest and fastest to scale once a spec is written.

This three-layer structure is the same shape referenced elsewhere on this site when discussing how to build a pretraining data mixture: a wide base of general, scalable data narrowing toward a small, embodiment-specific top layer. GR00T N1 is the paper that originally describes this structure for a humanoid vision-language-action model.

Most egocentric human video has no robot action label attached to it — there is no joint angle or gripper command recorded alongside the footage, because no robot was involved in capturing it. The GR00T N1 paper (arXiv 2503.14734) addresses this with latent actions and inverse-dynamics pseudo-labels: rather than requiring an explicit action label on every frame, the model learns a latent action representation from unlabeled video and uses an inverse-dynamics approach to infer action-relevant signal from the visual change between frames.

This matters for the base layer specifically, because it is what lets a wide base of unlabeled or human-only video still contribute action-relevant training signal, rather than being usable only for general visual pretraining. Without this technique, the base layer's contribution would be limited to visual priors; with it, the base layer also informs the model's action representation before any robot-specific data enters the pyramid at all.

The distinction between a latent action and an inverse-dynamics pseudo-label is worth separating out, since both address the same missing-label problem but work differently. A latent action is a learned, compressed representation of "what changed between two frames" that the model infers directly from video, without ever being told what the underlying action actually was. Inverse dynamics, by contrast, is the more explicit strategy of estimating the action that most plausibly explains an observed state transition — effectively working backward from the visual outcome to a probable cause. The GR00T N1 paper uses both together: the model does not require ground-truth action labels on the base layer to begin learning action-relevant structure from it.

Real robot data sits at the pyramid's narrow top because it is the only layer that captures the exact embodiment, actuator dynamics and sensor characteristics the deployed policy will actually see — and it is also the most expensive and slowest layer to collect at volume, since it requires a functioning robot, a teleoperation rig, and (usually) a human operator. Every other layer can be produced at greater scale without those constraints: web data and human video need no robot at all, and synthetic data needs a simulator rather than physical hardware.

The pyramid's shape reflects this cost-and-specificity tradeoff directly. A team could, in principle, try to pretrain entirely on real robot data, but the volume achievable that way is far smaller than what a human-video or synthetic base layer can supply — which is exactly why GR00T N1's pyramid uses real robot data sparingly, as the top alignment layer, rather than as the primary volume source.

The three pyramid layers map onto the three training stages covered elsewhere on this site, though not in a strict one-to-one way. Pretraining draws primarily on the base layer (web data and human video), with the middle synthetic layer often blended in during the same pretraining phase or a following mid-training phase to add robot-specific dynamics cheaply. Post-training then narrows almost entirely to the top layer: real robot demonstrations for the specific skills a deployed policy needs.

• Pretraining: leans on the base layer ��� web data and egocentric human video — for broad visual and manipulation priors
• Mid-training: often where the middle synthetic layer, and any small aligned real-robot slice, get introduced to bridge general pretraining toward the target embodiment
• Post-training: scoped almost entirely to the top layer of real robot demonstrations for the target skill set

EgoScale's published recipe (arXiv 2602.16710, NVIDIA GEAR) illustrates a similar layered approach in practice: a large 20,854-hour egocentric pretraining corpus followed by a much smaller, aligned mid-training slice — roughly 50 hours of human video paired with about 4 hours of matched robot demonstrations across 344 tabletop tasks. The shapes differ in scale but follow the same wide-base-to-narrow-top logic the GR00T N1 pyramid describes.

Because the base layer is the layer easiest to scale and diversify, it is also the layer where a team has the most control over what gets collected. Unlike real robot data, which requires a functioning robot before collection can even start, egocentric human video can be commissioned against a coverage plan — specific scenes, tasks, objects, people and camera viewpoints — well before a robot platform or simulator is ready.

This is also the layer where the latent-action and inverse-dynamics techniques described above matter most for sourcing decisions: because unlabeled video can still contribute action-relevant signal, a team does not need every frame of base-layer footage to carry an explicit robot action label to make it useful. What it does need is coverage — the scenes, tasks and viewpoints the model will eventually need to generalize across — and consistent capture quality (calibration, synchronization, and where available, hand pose or other action-adjacent annotation) so the video is usable regardless of which technique later extracts signal from it.

Latent actions and inverse dynamics let unlabeled video contribute to the base layer, but a base layer that is directly action-labeled — for example, with 3D hand pose or explicit action segments captured at the time of recording — gives the model a more direct supervisory signal than one relying purely on pseudo-labels inferred after the fact. This does not replace the pyramid's structure; it changes what the base layer is capable of teaching before the model ever reaches the synthetic middle layer or the real-robot top layer.

This is one reason egocentric human video collection increasingly captures hand pose, action segments and viewpoint metadata alongside the raw footage, rather than delivering unannotated video alone: it lets the base layer carry action-relevant signal directly, rather than depending entirely on techniques like inverse dynamics to infer it after collection.

Sourcing the base layer for a pyramid-style pretraining plan follows the same pilot-then-scale approach described elsewhere on this site for any pretraining collection: write a spec naming the target scenes, tasks, objects, people and camera viewpoints the base layer needs to cover, run a small pilot to validate calibration, synchronization and label quality, then scale into the full base layer once that pilot confirms quality.

A custom collection scoped this way lets a team target exactly the base-layer coverage its pyramid plan calls for — rather than relying solely on whatever public egocentric corpora happen to already exist — and can be sized against the diversity gaps a mixture plan identifies rather than against an arbitrary hour target.

A base-layer spec written this way typically names four things before any capture begins: the target scenes (which environments the model needs to generalize across), the target tasks (which manipulation or locomotion behaviors should appear in frame), the target objects (which items the model needs to have seen enough variations of), and the target people (enough demographic and body-size variation that the model does not overfit to a narrow set of hands or gaits). Reviewing a pilot batch against that same four-part spec, before committing to full-scale collection, is what keeps the base layer's eventual composition matched to what the pyramid plan actually called for.