Pretraining Data Collection for Robot Foundation Models

Robot pretraining data is the corpus at the base of a foundation model’s training, before it is specialized to any one robot or task. NVIDIA’s GR00T N1 describes this as a data pyramid: web data and human videos sit at the base because they are the most abundant, synthetic data sits in the middle, and real robot trajectories sit at the top because they are the scarcest and most expensive to collect.

That pyramid is why pretraining leans on human data rather than robot data: it is the layer that can realistically be captured at the scale a foundation model needs, before the smaller, targeted stages that follow it. A pretraining corpus is judged on breadth first — how many distinct skills, objects, and environments it spans — because that breadth is what later, narrower stages build on.

At the base of the GR00T N1 pyramid sit web-scale data and human video: footage of people performing everyday and skilled manual tasks, captured from a viewpoint close to what a robot’s onboard camera would see. Synthetic data occupies the middle tier, used to add coverage and variation that would be slow to capture by hand, and real robot trajectories sit at the top, where they are the most expensive per hour and reserved for the mid-training and fine-tuning stages that follow pretraining.

Within the human-video tier, what matters is not raw footage but footage that carries a learnable signal: which hand is doing what, when contact happens, and what action is underway. That is why the pretraining corpora referenced throughout this page �� EgoScale, EgoVerse, Egocentric-10K, GEN-0, and Figure’s Project Go-Big — are described in terms of hours of action-labeled or annotated video, not just raw hours of recording.

EgoScale documents a two-stage recipe that mirrors how the field trains these models generally: large-scale pretraining on human data first, then a lightweight, aligned human-robot mid-training stage that adapts the pretrained representation to a target embodiment, before task-specific fine-tuning on the deployment robot.

Each stage narrows scope and raises cost per hour. Pretraining is broad and human-sourced; mid-training is a smaller, closely matched dataset aligned to the target robot’s embodiment; fine-tuning or post-training is scoped to the exact deployment task. EgoScale’s own mid-training stage illustrates the ratio: its 20,854-hour human pretraining corpus is followed by roughly 50 hours of human video plus about 4 hours of robot data across 344 tabletop tasks — a small, tightly aligned dataset compared with the pretraining layer beneath it.

Egocentric human video gives a foundation model the same first-person viewpoint and physically grounded motion a robot will act from, without needing a robot on hand to record it. That is why groups pursuing scale, such as Generalist’s GEN-0 program, describe accumulating hundreds of thousands of hours of real-world manipulation data and adding more than 10,000 new hours every week.

Generalist’s own framing is worth repeating: "data quality and diversity matters more than sheer volume." Scale alone is not the point; scale across enough distinct tasks and settings is. That is also why egocentric human capture, rather than robot teleoperation, is the layer that scales: a person wearing a capture rig can move through many more tasks, rooms, and objects per hour than a teleoperated robot arm can, at a fraction of the hardware cost.

EgoScale reports a log-linear scaling law between the amount of human pretraining data and validation loss, one that correlates with real-robot task performance downstream. On a 22-degree-of-freedom dexterous hand, the paper reports a 54% average improvement in success rate over training with no human pretraining at all.

That result comes specifically from the two-stage recipe, not from human pretraining alone: the 20,854-hour pretraining corpus establishes the broad prior, and the aligned mid-training stage — roughly 50 hours of human video and about 4 hours of robot data across 344 tabletop tasks — is what carries that prior onto the 22-DoF hand where the 54% figure was measured.

Other groups are converging on the same bet at meaningfully different scales: EgoVerse reports roughly 1,362 hours across about 80,000 episodes, Egocentric-10K is a 10,000-hour egocentric dataset released under an Apache 2.0 license, and Figure’s Project Go-Big and HumanScale are both pursuing large-scale human data collection for the same pretraining purpose. See the full comparison table below for how these corpora line up on hours, capture setup, and license.

Volume by itself is not the signal. Generalist’s own conclusion from GEN-0 is that data quality and diversity across tasks matter more than sheer hour counts, and EgoScale’s scaling law was measured against action-labeled data, not raw unlabeled footage.

That means a useful pretraining corpus needs synchronized hand pose and action segments so the video carries a learnable signal, hardware-aligned streams so fused sensors are trustworthy at scale, and licensing that survives a commercial training run rather than research-only terms.

It also means the way labels are produced matters. EgoScale’s action labels come from off-the-shelf SLAM and hand-pose estimation applied after the footage is captured — a method that scales to tens of thousands of hours but estimates pose and contact rather than measuring them directly on a synchronized rig. A corpus with manipulation-grade 3D hand pose and depth captured live, on the same hardware clock as the video, carries a cleaner signal per hour than one where those signals are reconstructed afterward.

This is the same distinction that separates imitation learning from a hand-designed reward function: a policy trained by imitation learns directly from demonstrated trajectories, so its ceiling is set by demonstration quality — viewpoint, coverage, and label fidelity — not by the reward engineer’s intuition. A pretraining corpus feeding that kind of policy inherits the same ceiling. Action segments and hand pose are the signals an imitation-learning or behavioral-cloning policy consumes directly, which is why they matter more to usefulness than an extra thousand hours of unlabeled footage would.

The public corpora above differ mainly on how they were captured and what they license, not on whether scale is achievable. EgoScale’s action labels come from off-the-shelf SLAM and hand-pose estimation rather than a purpose-built rig, and most large egocentric corpora carry non-commercial or research-only terms rather than a right to train a commercial product on them.

A custom pretraining collection closes both gaps at once: it is captured to a written spec across the tasks and environments a model is actually missing, with hardware-synchronized streams and manipulation-grade pose throughout, under a license written for commercial training from the outset. It also closes a third gap public corpora cannot: coverage. A public dataset was captured for someone else’s task distribution, so it will always be missing exactly the skills, objects, or environments a given model needs next — a custom collection is scoped to fill that specific gap rather than to be generally useful.

None of this makes public pretraining corpora unusable — EgoScale’s own results show a research-grade corpus, action-labeled by off-the-shelf SLAM and pose estimation, is enough to demonstrate a real scaling law and a measurable success-rate gain. The choice between public and custom data is about what a given program needs next: broad, general priors are exactly what large public corpora are built for, while filling a specific, known gap in coverage or clearing commercial licensing terms is what a custom collection is built for.

The same capture stack used across every Firsthand program, scoped broadly for pretraining rather than to one deployment task: ten synchronized streams per episode — 4K head camera, two wrist cameras, depth, IMU, 3D hand and body pose, instance segmentation, and action labels — hardware-aligned under 2 ms.

Coverage is set by a written spec across tasks, environments, and regions, so a pretraining corpus can span the skill and setting diversity a foundation model needs rather than one recorded location. Because hand pose, depth, and action labels are captured live on the rig rather than estimated from raw video afterward, every episode carries the same manipulation-grade signal a mid-training or fine-tuning batch would, at pretraining scale.

Mid-training needs a smaller, more targeted dataset than pretraining: episodes scoped to the specific embodiment and task family a model will be aligned to next, rather than the broad spread a pretraining corpus covers. The same synchronized stream stack and accept criteria apply; only the spec narrows.

Because mid-training and pretraining run on the same underlying capture pipeline, a program can move from broad pretraining coverage to a closely aligned mid-training batch without switching data providers or formats. That continuity matters when a model has already pretrained on a broad corpus and now needs a smaller, embodiment-matched dataset to bridge into deployment — the exact role EgoScale’s ~50-hour human plus ~4-hour robot mid-training stage plays ahead of its 22-DoF hand evaluation.

The ratio in that recipe is worth noting: a pretraining corpus tens of thousands of hours deep is followed by a mid-training stage measured in dozens of hours, not thousands. That is what "lightweight" means in the EgoScale recipe — mid-training does not need to rival pretraining in scale, it needs to be closely matched to the target embodiment and camera geometry so the pretrained prior transfers cleanly onto the deployment robot.

You describe the skill and environment diversity your pretraining corpus is missing. That becomes a written spec covering tasks, environments, regions, capture tier, annotation depth, and accept criteria, quoted for reach, timeline, and price before any capture begins. Many programs start with a Spec Pilot, a small, fast first batch that proves the spec and accept criteria before scaling volume.

From there the collection runs through the same seven-stage end-to-end pipeline as every Firsthand program: contributor recruiting, capture, per-batch QA, anonymization, and licensed delivery, with rejected items recaptured at our cost and on-demand top-ups running against the same accept criteria afterward. A median 48-hour turnaround on a first batch means a pretraining program can validate its spec quickly, then scale the same pipeline to the volume the model needs.

Because that pipeline is the same one used for every Firsthand collection — not a separate process for pretraining volume — a program gets the same per-batch QA and accept-criteria enforcement at pretraining scale that a smaller mid-training or fine-tuning batch would get. Volume does not trade off against the quality bar each episode has to clear before it ships.

Every custom collection ships under a perpetual, buyer-owned commercial license, with exclusivity available so the data is never resold, and each episode links to a signed contributor release through a documented consent chain — the piece many public pretraining corpora do not clear for commercial use.

Faces, plates, and screens are detected and irreversibly blurred at ingest, and a reviewer confirms the result before anything ships. The anonymization record is delivered with the data, so a pretraining program has a clear paper trail from contributor consent through anonymization to the license terms governing commercial training on the finished corpus.

RLDS, LeRobot, and WebDataset, plus HDF5, zarr, and Rerun .rrd, so a pretraining corpus loads directly into the same pipelines used for mid-training and fine-tuning data. Converter source is readable, so you can retarget the schema to your own loader. Because pretraining, mid-training, and fine-tuning data all ship through the same delivery formats, a program does not need a separate ingestion pipeline for each training stage, whether the corpus is ten thousand episodes deep or a small aligned mid-training batch.

Start with the gap in your current pretraining corpus rather than the total volume you think you need: which tasks, objects, or environments is the model still weak on. That gap becomes the written spec — tasks, environments, regions, capture tier, annotation depth, and accept criteria — quoted for reach, timeline, and price up front.

A Spec Pilot proves that spec on a small, fast first batch before any commitment to scale, so you can check stream quality, hand-pose and action-label fidelity, and format compatibility with your training pipeline before volume ramps. From there, the same seven-stage pipeline scales the corpus to the size the model needs, with on-demand top-ups available afterward as new gaps in coverage show up during training.