Training Data for Robot Foundation Models

A robot foundation model is trained in stages, and each stage runs on a different kind of data. NVIDIA’s GR00T N1 describes the underlying resource as a data pyramid: web data and human videos sit at the base because they are the most abundant and cheapest to capture at scale, synthetic data sits in the middle, and real robot trajectories sit at the top because they are the scarcest and most expensive.

That pyramid maps onto the three training stages a foundation model actually runs through: broad pretraining draws mostly from the base of the pyramid (human video), mid-training blends a small amount of matched human and robot data, and post-training (fine-tuning) draws almost entirely from the top — real robot demonstrations of the specific deployment task.

Pretraining data is broad and human-sourced: egocentric video, depth, hand pose, and action labels captured across many tasks, objects, and environments, with no specific robot in mind. EgoScale’s published recipe uses a 20,854-hour human pretraining corpus as this base layer.

Mid-training data is a smaller, embodiment-specific dataset that pairs matched human and robot capture, aligned in viewpoint and calibration to one target robot. EgoScale’s recipe pairs roughly 50 hours of human video with about 4 hours of robot data across 344 tabletop tasks — dozens of hours, not thousands.

Post-training (fine-tuning) data is the smallest and most specific stage: robot demonstrations of the exact task the deployment robot will run, scoped narrower still than mid-training. Each stage narrows scope and raises cost per hour as it goes.

Egocentric human video gives a model the same first-person viewpoint and physically grounded motion a robot will act from, without needing a robot on hand to record it. A person wearing a capture rig can move through far more tasks, rooms, and objects per hour than a teleoperated robot arm can, which is why the pretraining layer of the pyramid scales on human data rather than robot data.

Generalist’s GEN-0 program describes accumulating hundreds of thousands of hours of real-world manipulation data and adding more than 10,000 new hours every week, with the explicit framing that "data quality and diversity matters more than sheer volume." Other groups pursuing pretraining scale at meaningfully different sizes include EgoVerse (roughly 1,362 hours across about 80,000 episodes) and Egocentric-10K (a 10,000-hour egocentric dataset released under an Apache 2.0 license), alongside Figure’s Project Go-Big and HumanScale, both pursuing large-scale human data collection for the same pretraining purpose.

The same capture stack runs across all three stages: ten synchronized streams per episode — 4K head camera, two wrist cameras, depth, IMU, 3D hand and body pose, instance segmentation, and action labels — hardware-aligned under 2 ms. What changes between stages is scope and specificity, not stream count or label fidelity.

For pretraining, coverage is set by a written spec across tasks, environments, and regions. For mid-training, capture pairs a human contributor with the target robot on the same calibrated rig, so both streams share the same camera intrinsics. For post-training, capture is scoped to the exact task list the deployment robot will run. Delivery is consistent across all three: RLDS, LeRobot, WebDataset, or HDF5, under a perpetual, buyer-owned commercial license.

Diversity is scoped by a written spec rather than collected as a generic pile of footage. A pretraining spec can stay broad — general household or workplace coverage, a wide task and object mix — because the goal is a general prior. A mid-training or post-training spec has to name the target robot’s specific camera geometry, reach, and task list, because every episode is measured against how closely it matches that embodiment.

Across all three stages, the same manipulation-grade capture applies: hand pose, contact events, and action segments are captured live on the rig, so diversity in tasks and settings does not come at the cost of label quality.

Many large public egocentric corpora ship under research-only or non-commercial terms, which is a real constraint for a program that intends to train a commercial foundation model rather than publish a paper. A custom collection across pretraining, mid-training, and post-training data is captured with documented contributor consent and delivered under a perpetual, buyer-owned commercial license from the outset, so licensing does not become a blocker partway through a training run.

Egocentric-10K is a notable exception among public corpora, released under an Apache 2.0 license — but it, like other public datasets, was captured for someone else’s task distribution, so it will not cover the specific skills, objects, or environments a given model needs next.

The licensing question matters more at mid-training and post-training than at pretraining, because those stages capture data on or alongside an actual robot rather than a person acting on their own. A written spec for either stage should say up front how the resulting data may be used — for training only, for evaluation, for redistribution to a partner — so the license terms match the program’s actual plans rather than being renegotiated after capture is already underway.

A program starts with a written spec naming the training stage (pretraining, mid-training, or post-training), the target task and environment diversity, and, for mid-training or post-training, the target robot’s camera geometry and reach. First batches typically arrive at a median of 48 hours from signed spec where coverage is deep, or three to four weeks where coverage is thin and contributors need to be recruited first.

Because pretraining, mid-training, and post-training data all run on the same synchronized capture pipeline, a program can move between stages — or run more than one at once — without switching data providers, formats, or licensing terms. A team that already has a broad pretraining corpus can scope a mid-training-only collection aligned to its specific robot; a team starting from nothing can scope all three stages against one written spec.

A spec typically fixes the variables that matter for the stage in question and leaves the rest open for the collection team to fill in against the target distribution. For pretraining, that usually means naming the environment categories (kitchens, warehouses, retail floors, offices) and the task families within each, while leaving the exact objects and room layouts to natural variation across contributors and sites. For mid-training and post-training, the spec instead fixes the robot’s camera geometry, reach envelope, and gripper or hand configuration, then varies the task and object set within that fixed embodiment.

A program that has no existing data at all typically starts with pretraining, since a broad human-behavior prior gives every later stage something to build on — without it, mid-training and post-training data have nothing general to specialize. A program that already has a broad pretraining corpus, whether public or previously collected, but has never run on a specific robot can skip straight to a mid-training collection scoped to that robot’s camera geometry.

A program with a working policy that already performs reasonably well in pretraining and mid-training, but is failing on a narrow set of deployment tasks, usually needs post-training data rather than more of either broader stage — collecting more pretraining hours will not fix a policy that has not seen the specific task, object, or edge case it keeps failing on. Because all three stages share the same capture pipeline and delivery formats, a program is free to run stages in whatever order or combination its current data gap calls for.