What Is a Data Pyramid in Robot Learning?

No single data source supplies both the scale and the embodiment-specific accuracy a robot foundation model needs, which is why the pyramid framing exists: it makes explicit that each layer trades volume for specificity, rather than treating "more data" as one undifferentiated pool. NVIDIA’s GR00T N1 paper reports using latent actions and inverse-dynamics pseudo-labels to make its unlabeled base-layer video — which carries no native robot action labels — usable for training anyway, rather than discarding it for lacking labels a robot demonstration would have. That design choice is also why the pyramid is a planning tool, not just a description: a team can see which layer is thin and go fill it, instead of adding undifferentiated hours to a layer that is already well covered.

Firsthand captures the synchronized egocentric video, depth, 3D hand pose, and action-segment data that populate a pyramid’s base layer, collected fast, on demand, and to a written spec, then delivered in RLDS, LeRobot format, or WebDataset with a buyer-owned commercial license. Because the base layer’s value comes from coverage — scenes, tasks, objects, and people — rather than any single volume target, a custom collection can be scoped around exactly the gaps a public corpus or an existing mixture does not already fill.

A data pyramid is a way of structuring the data that goes into a robot foundation model’s training run by stacking sources from broadest to narrowest, rather than treating them as one undifferentiated pool. NVIDIA’s GR00T N1 paper (arXiv 2503.14734) describes training on a pyramid with a wide base of web data and egocentric human video, a middle layer of synthetic data, and a narrow top layer of real robot trajectories.

The shape of the pyramid — wide at the bottom, narrow at the top — reflects a tradeoff: the base layer is cheap to scale and broadly diverse but furthest from the target robot’s exact embodiment, while the top layer is closest to that embodiment but far smaller and more expensive to collect.

In the GR00T N1 paper’s framing, the base layer is web-scale data and egocentric human video: the highest-volume, most general source, useful for broad visual and task coverage but carrying no direct robot action labels. The middle layer is synthetic or simulated data, generated to add robot-specific scenarios at a volume still well above what physical capture alone could provide. The top layer is real robot trajectory data on the target hardware — the smallest layer by volume, but the only one that captures the exact actuator and sensor characteristics of the deployment robot.

See the table below for how the layers compare on volume and labeling.

NVIDIA’s GR00T N1 paper (arXiv 2503.14734) trains its vision-language-action model across this layered structure rather than on a single dataset. A key design detail the paper describes is how it makes the unlabeled base-layer video useful despite lacking native robot action labels: it uses latent actions and inverse-dynamics pseudo-labels to recover action-relevant signal from that video, so the base layer still contributes to action learning rather than only to perception. The GR00T N1 data pyramid, explained walks through the paper’s pyramid and labeling approach in full.

A data pyramid is the specific layered shape GR00T N1’s paper describes — base, middle, and top, ordered by volume and embodiment closeness. A data mixture is the broader, more general term for the blend of sources and their proportions that go into any pretraining run, whether or not that blend is organized into pyramid layers. Every data pyramid is a data mixture, but not every data mixture is laid out as a pyramid.

Egocentric human video sits in the base layer, alongside web data, as the highest-volume and most general source in the pyramid. It is first-person footage recorded from a camera worn by the person performing a task, which gives it the viewpoint and hand-object contact a robot’s own head or wrist camera will see — but, like other base-layer data, it lacks native robot action labels, which is why GR00T N1’s paper applies latent actions and inverse-dynamics pseudo-labels to make it trainable for action learning rather than perception alone.

A custom collection can be scoped around exactly which scenes, tasks, objects, or people the base layer is missing, rather than adding undifferentiated volume to a layer that already has broad coverage. Firsthand captures that base-layer egocentric video — synchronized RGB, depth, 3D hand pose, and action-segment labels — fast and on demand, to a written spec, and delivers it in RLDS, LeRobot format, or WebDataset under a buyer-owned commercial license.

What is a data pyramid in robot learning?

A data pyramid is a way of organizing robot training data by volume and embodiment closeness: a wide base of general data such as web data and egocentric human video, a middle layer of synthetic data, and a narrow top layer of real robot data for the target embodiment, as described in NVIDIA’s GR00T N1 paper.

Who introduced the robot data pyramid?

NVIDIA’s GR00T N1 paper (arXiv 2503.14734) describes training its vision-language-action model on this layered structure.

What is at the base of the data pyramid?

Web data and egocentric human video — the highest-volume, most general layer, used for broad coverage rather than embodiment-specific accuracy.

What is at the top of the data pyramid?

Real robot trajectory data captured on the target hardware — the smallest layer by volume, but the only one with the exact actuator and sensor characteristics of the deployment robot.

How does GR00T N1 use unlabeled video in its data pyramid?

The GR00T N1 paper applies latent actions and inverse-dynamics pseudo-labels to its base-layer video, recovering action-relevant signal even though that video carries no native robot action labels.

Is a data pyramid the same thing as a data mixture?

No. A data mixture is the general term for any blend of sources and proportions in a training run. A data pyramid is the specific layered shape — base, middle, top — that GR00T N1’s paper describes; every data pyramid is a data mixture, but not every mixture is a pyramid.

Where does custom egocentric collection fit into a data pyramid?

Custom egocentric collection fills the base layer: it is scoped around the specific scenes, tasks, objects, or people a public corpus or existing mixture does not already cover, rather than adding undifferentiated volume to a layer that is already broad.

Does the data pyramid apply outside GR00T N1?

The three-layer pyramid is specific to how NVIDIA’s GR00T N1 paper describes its own training data. Other published efforts, such as GEN-0, describe their own mixtures and scaling approaches without necessarily using the same pyramid framing — see what is a data mixture for the more general concept.