Humanoid Robot Pretraining Data

A humanoid foundation model learns a general bimanual and whole-body prior during pretraining, before fine-tuning specializes it on any one task. That takes egocentric video captured from a head-worn viewpoint, with both hands and whole-body pose in frame, spanning enough homes, workplaces, tasks, and people that the model does not overfit to one narrow setting.

This is a different requirement from fine-tuning, where a small, task-specific batch specializes an already-pretrained humanoid model. Pretraining data is judged on breadth and diversity; fine-tuning data is judged on how precisely it represents the target task and embodiment. See pretraining vs mid-training vs post-training data for robots for the full three-stage breakdown.

A humanoid with a head-mounted camera occupies almost exactly the viewpoint a head-worn capture rig records. Egocentric human video matches that observation space far more closely than third-person video or single-arm teleoperation, which fix a different camera position and a different action space, and it can be captured at far greater scale than teleoperated robot data.

NVIDIA’s EgoScale paper reports pretraining a robot foundation model on a 20,854-hour corpus of action-labeled egocentric human video, and found a log-linear scaling law between egocentric pretraining data volume and downstream task success — with a 54% success-rate improvement on a real 22-DoF robotic hand after adding a small amount of robot fine-tuning data on top. That result is about egocentric human video used for pretraining generally, not specific to any one embodiment.

Figure’s Project Go-Big, described on Figure’s official project page, is a stated commitment to scaling humanoid data collection as part of training its humanoid robots — evidence that leading humanoid developers treat data volume and diversity as a primary lever, not a secondary concern.

NVIDIA’s GR00T N1 paper describes training its humanoid foundation model on a pyramid of data sources at different scales and levels of realism, from broad web and human video data at the base to real robot trajectory data at the top. Egocentric human video sits in the middle of that pyramid — more physically grounded than web video, more scalable to collect than real robot teleoperation.

Two wrist cameras plus 3D hand pose keep both hands in frame with per-hand pose and visibility flags throughout coordinated, often asymmetric two-handed work. Body pose and IMU add the whole-body context: stance, reach, and the transfer between locomotion and manipulation that a humanoid task depends on.

Instance segmentation and action segments mark what each hand is doing and when, so a policy can learn the handoff between the two — the same label families used across Firsthand’s action-labeled egocentric video and humanoid robot collections.

A pretraining spec sets targets for environments (homes, workplaces, outdoor settings), task categories, object sets, and contributor demographics before capture begins, so the resulting corpus does not overfit to one kitchen, one task, or one body type. That target list becomes the written spec every custom collection is quoted against.

This is the same diversity-first planning approach described in do robot foundation models follow data scaling laws?, where EgoScale’s and GEN-0’s findings both point to composition mattering as much as total hours.

Pretraining establishes the general bimanual and whole-body prior across a wide range of homes, tasks, and people. Mid-training narrows that broad prior toward the humanoid’s specific embodiment and target domains, using a more curated — but still diverse — dataset than pretraining, before fine-tuning specializes on one exact task. The full three-stage comparison, including volumes and capture requirements at each stage, is covered in pretraining vs mid-training vs post-training data for robots.

Ten synchronized streams per episode: a 4K head camera, two wrist cameras, depth, IMU, 3D hand and body pose, instance segmentation, and action labels, hardware-aligned under 2 ms so every stream sits on one clock. That is the same capture stack used across custom training data for humanoid robots, applied at pretraining breadth rather than a single task’s scope.

Every collection ships under a perpetual, buyer-owned commercial license, with faces, plates, and screens irreversibly blurred at ingest and a documented consent chain behind every episode.

A spec written around hour count alone, without explicit environment, task, and contributor targets, tends to converge on whatever is easiest to capture repeatedly: the same room, the same few tasks, a narrow contributor pool. That produces a corpus that looks large by the hour but behaves like a much smaller one, because a foundation model pretraining on it sees the same visual and motor patterns over and over rather than the breadth it needs to generalize.

This is the practical reason a written spec sets diversity targets before capture begins rather than after: it is far cheaper to correct a coverage gap in the spec than to discover it in a trained model’s failure modes months later. EgoScale’s and GEN-0’s own published emphasis on diversity mattering as much as hours reflects the same lesson at a larger scale.

Start with the environments, task categories, and contributor diversity your pretraining stage needs to cover. That becomes a written spec, quoted for reach, timeline, and price before capture begins.

Many programs start with a Spec Pilot — a small, fast first batch that proves coverage and label quality before scaling volume — then run through the same seven-stage pipeline used for every Firsthand collection, with a median 48-hour first validated batch where coverage is deep, delivered under a buyer-owned perpetual license in the same formats used across every Firsthand collection.