What Is Robot Pretraining Data?

Robot pretraining data is the first, largest layer of data a robot foundation model learns from. Its job is not to teach a specific task on a specific robot — it is to teach general priors: what hands look like as they move through the world, how objects respond to contact, what a kitchen or a warehouse looks like from a first-person viewpoint, and how actions unfold over time. Because this layer does not need to come from the target robot itself, it can be collected at a scale that would be impractical with only robot teleoperation.

This is why egocentric human video — video captured from a head- or chest-worn camera as a person performs real tasks with their own hands — has become a leading pretraining source for robot foundation models. It captures the same kind of first-person, hand-centric, action-rich footage a robot’s onboard camera would see, at a scale and cost human demonstration data can reach that teleoperated robot data typically cannot.

Robot foundation models are generally trained in stages, and the terminology matters because each stage uses a different kind of data for a different purpose.

The NVIDIA EgoScale research describes this as a two-stage recipe: large-scale human pretraining followed by lightweight, aligned human-robot mid-training. The pretraining stage does the heavy lifting on scale; the mid-training stage is comparatively small but does the work of bridging human hands and human viewpoints to the specific robot embodiment and camera rig the model will actually run on.

Firsthand supports this exact split: broad egocentric human-video capture for the pretraining layer, and aligned mid-training capture — matching camera position, calibration and task framing to a target robot rig — for the mid-training layer. See how a pretraining collection engagement works below.

Public research describes robot foundation model pretraining as drawing on several data types at once, not a single source. NVIDIA’s GR00T N1 paper frames this as a "data pyramid": web-scale data and human video sit at the base because they are the most abundant, synthetic data sits in the middle to fill gaps and add controlled variation, and real robot data sits at the top because it is the most expensive to collect but the most directly relevant to the deployed embodiment.

Egocentric human video sits at the base of that pyramid because it is both abundant and close to the robot’s actual sensory experience: a first-person, hand-centric viewpoint of a person completing real tasks. That combination — scale plus a viewpoint that resembles a robot’s own cameras — is what makes it different from generic third-person internet video for this purpose.

EgoScale, published by NVIDIA’s GEAR lab, is one of the most direct public studies of egocentric human video as robot pretraining data. The researchers assembled 20,854 hours of action-labeled egocentric human video, with action labels derived from off-the-shelf SLAM and hand-pose estimation rather than manual annotation, and used it to pretrain policies before a lightweight aligned mid-training stage on a small amount of real robot data.

Two findings from that work are widely cited for exactly the question this guide answers. First, EgoScale reports a log-linear scaling law between the amount of human pretraining data and validation loss, and that this scaling correlates with real-robot downstream performance — in other words, more egocentric human video during pretraining kept producing measurable gains, not diminishing returns that plateaued early. Second, on a 22-degree-of-freedom dexterous hand, the two-stage human-pretraining-plus-mid-training recipe produced a 54% average success improvement over skipping pretraining entirely.

Both findings point the same direction: pretraining data scale is not a minor input to a robot foundation model — it is one of the levers with the clearest measured effect on downstream task success, at least for the dexterous manipulation setting EgoScale studied.

NVIDIA’s GR00T N1 paper describes training a generalist humanoid robot foundation model on a mixture of data types arranged like a pyramid by abundance and specificity. Web data and human video make up the wide base — plentiful, cheap relative to robot data, and useful for general visual and physical priors. Synthetic data occupies the middle layer, used to add variation and coverage that is hard or expensive to capture in the real world. Real robot data sits at the narrow top — the most expensive and most directly aligned with the deployed embodiment, and typically the smallest slice by volume.

The pyramid framing is useful because it explains why pretraining and mid-training play different roles: the base layer teaches breadth cheaply, and the top layer teaches precision on the target robot, using far less data than would be needed if the top layer had to do all the learning alone.

Generalist AI’s GEN-0 describes training on 270,000 hours of real-world manipulation data, growing by more than 10,000 new hours per week, and states directly that "data quality and diversity matters more than sheer volume" — a caution against reading pretraining-data scale as the only variable that matters. Scale helps, but only when the data is diverse and well-labeled; a large pool of narrow, repetitive footage does not substitute for coverage across environments, objects and failure modes.

HumanScale is a separate line of published research specifically studying the scaling behavior of human-video pretraining for robot policies, adding to the EgoScale evidence that this is now an active, evidence-backed area of robotics research rather than a single lab’s result.

Three properties make egocentric human video attractive specifically for pretraining, as opposed to mid-training or post-training, where footage from the actual target robot still matters most.

• Viewpoint match: a head- or chest-worn camera produces a first-person, hand-centric view close to what a robot’s onboard cameras see, unlike third-person internet video.
• Collection scale: human demonstrators can perform real tasks across many more environments, objects and conditions than a teleoperated robot rig can practically reach.
• Action structure: modern pipelines can derive action labels — hand pose, contact events, task segments — from off-the-shelf estimation methods, as EgoScale did, rather than requiring the video to already be a robot trajectory.

None of this replaces mid-training or post-training on the target robot — the EgoScale recipe explicitly pairs large-scale human pretraining with a lightweight, aligned mid-training stage on real robot data. It explains why human video is the layer that scales, while robot-specific data remains the layer that specializes.

Scale alone is not the whole story — GEN-0’s own framing makes that explicit. For pretraining data to translate into downstream robot performance, several properties matter alongside hours captured.

• Diversity across environments, objects, tasks and demonstrators, so the model does not overfit to one kitchen, one grasp style or one lighting condition.
• Action labels — hand pose, contact events and task segmentation — so the video carries structure a policy can learn from, not just pixels.
• Camera calibration, so intrinsics and extrinsics are known and downstream depth or pose estimation from the footage is reliable.
• Cross-stream synchronization, so multiple camera or sensor streams captured together stay aligned in time and can be fused rather than treated as loosely related recordings.

Firsthand’s capture stack is built around exactly these properties: ten streams captured in hardware-triggered sync under 2 ms, calibrated stereo cameras, and 3D hand pose with contact events derived at capture time — the same categories EgoScale used off-the-shelf estimation to derive after the fact.

There is no single published answer to "how much pretraining data is enough" — the honest reference points are the volumes public research groups have reported using, which vary by an order of magnitude depending on the study and the task scope.

What these reference points share is not a specific hour target but a pattern: teams pretraining robot foundation models are working in the thousands-to-hundreds-of-thousands-of-hours range for the human-video layer, several orders of magnitude beyond what teleoperated robot data collection alone typically produces. The right number for any given model depends on task scope, embodiment complexity and how much mid-training and post-training data will follow — which is why a custom pretraining collection engagement typically starts with a small pilot batch rather than committing to a large volume up front.

Public corpora are a useful starting point and a valid reference for what "enough" has looked like elsewhere, but they are fixed: the environments, tasks, object sets and camera rigs were chosen by whoever collected them, not by the team building a specific robot foundation model. Custom pretraining collection exists for teams that need coverage a public corpus does not have — a specific task distribution, a specific environment type, or a specific camera and calibration setup that matches a target robot’s eventual sensor stack.

Aligned mid-training capture is the bridge step between broad pretraining data and a specific robot. Rather than general egocentric footage across many tasks and environments, mid-training capture narrows in on the target embodiment: camera position and calibration matched to the robot’s eventual sensor rig, task framing that mirrors what the deployed robot will actually attempt, and enough coverage to align the general priors learned during pretraining to that specific setup — without needing anywhere near as much volume as the pretraining layer itself, consistent with EgoScale’s description of mid-training as "lightweight."

A custom pretraining data engagement with Firsthand follows the same process used across all custom collection work, scaled to pretraining volumes.

• Spec Pilot: define the task distribution, environments and any embodiment-specific framing needed, and validate the spec against a small pilot batch before committing to scale.
• Capture at scale: broad egocentric human-video collection for the pretraining layer, or aligned mid-training capture matched to a target robot rig, run through the same seven-stage pipeline used for all Firsthand data.
• Validation and delivery: episodes are validated against the agreed spec and delivered in the buyer’s chosen format, on a per-validated-hour billing model with a buyer-owned perpetual license.

This is the same collection model described in full on the pretraining data collection page, applied specifically to the pretraining and mid-training layers described in this guide.

Licensing matters as much for pretraining data as for any other training data layer, and the same distinction applies: a research-only or non-commercial license on a public pretraining corpus does not clear the way to train a model that will later be deployed commercially. Pretraining a model that will eventually ship as a commercial product is generally treated as a commercial use of that data regardless of how early in the training pipeline it was used, so the license terms of any public corpus should be checked against the model’s eventual commercial use, not just its research origin.

Custom pretraining collection avoids this ambiguity by design: data collected under a buyer-owned perpetual license, with consent and anonymization handled as part of the pipeline, carries no downstream restriction on commercial use of the resulting model.

Pretraining data is delivered in the same formats used across Firsthand’s custom collection work: RLDS, LeRobot, WebDataset or HDF5, with an episode schema of meta, streams, annotations and calibration groups underneath each format. That consistency matters for pretraining specifically because a pretraining corpus is often mixed with mid-training and post-training data downstream, and a shared episode schema across all three layers keeps that mixture tractable instead of requiring a separate conversion step for each source.

A pretraining program that plans its delivery format up front, rather than converting after the fact, avoids a common failure mode: a corpus that scores well on its own metrics but does not load cleanly into the loader a mid-training or fine-tuning stage already depends on. Choosing the format alongside the capture spec, not after collection finishes, keeps every downstream training stage on the same schema.