Humanoid Robot Mid-Training Data

Humanoid mid-training data is the second stage in a three-stage training recipe for humanoid foundation models: broad pretraining on diverse human egocentric video first, then a smaller, aligned human-humanoid mid-training stage that adapts the pretrained bimanual and whole-body prior to one target embodiment, before task-specific fine-tuning (post-training) on the deployment humanoid.

It sits between humanoid pretraining and post-training in both scale and specificity. Pretraining is collected broadly, across many homes, workplaces, tasks, and people, with no single humanoid in mind. Mid-training is collected with one target humanoid’s head camera geometry, reach, and gripper already fixed, so every episode is directly comparable to what that humanoid will see and do.

A humanoid foundation model is not trained on one dataset; it moves through three. Pretraining establishes a general bimanual and whole-body prior across a wide range of environments and people. Mid-training then narrows that broad prior toward the humanoid’s specific embodiment — its head camera, arm reach, and gripper — using a more curated but still diverse dataset than pretraining. Fine-tuning (post-training) narrows further still, to one exact deployment task.

Each stage narrows scope and raises cost per hour as it goes: pretraining spans many skills and settings at the lowest cost per hour, mid-training aligns to one humanoid, and fine-tuning aligns to one task on that humanoid. The full three-stage breakdown, including how the stages differ in goal and typical data, is covered in pretraining vs mid-training vs post-training data for robots.

A humanoid pretrained on broad egocentric human video has learned general priors about two-handed manipulation, object interaction, and balance, but it has not learned the specific head camera geometry, field of view, or reach of any one humanoid. Mid-training closes that gap by recording from a viewpoint and calibration that match the target humanoid as closely as possible.

This matters more for humanoids than for simpler single-arm robots, because a humanoid’s observation space spans two wrist cameras, a head camera, and whole-body pose at once. Aligning all of that to one embodiment’s geometry — not just one camera — is what a humanoid mid-training collection has to deliver.

Alignment has four concrete parts. Viewpoint match means capture is recorded from a head-worn position and field of view that mirrors the target humanoid’s own head camera, not a third-person or overhead shot. Calibrated intrinsics means the camera’s lens distortion, field of view, and mounting offset are measured and matched to the target humanoid’s specifications before capture begins, so the geometry the model sees in mid-training is the geometry it will see at deployment.

Sync means every stream — head camera, two wrist cameras, depth, IMU, 3D hand and body pose — shares a single hardware clock, so human and humanoid episodes stay frame-comparable. Action labels means hand pose, contact events, and action segments are captured live during the episode rather than estimated afterward, matching the manipulation-grade signal used at pretraining scale. All four have to hold together; matching viewpoint alone without calibrated intrinsics, or syncing streams without action labels, leaves the alignment incomplete.

EgoScale documents a two-stage recipe: large-scale pretraining on a 20,854-hour human egocentric corpus, followed by a lightweight, aligned human-robot mid-training stage. That mid-training stage pairs roughly 50 hours of human video with about 4 hours of robot data, collected across 344 tabletop tasks (arXiv 2602.16710; NVIDIA GEAR Lab project page).

The 54% success improvement EgoScale reports on a 22-DoF hand comes from the full two-stage recipe, not from pretraining alone. The pretraining corpus establishes the broad prior; the aligned mid-training stage is what carries that prior onto the specific hand it was evaluated on. EgoScale’s published numbers are the clearest reference point available for how small a well-aligned mid-training set can be relative to a pretraining corpus — dozens of hours against tens of thousands.

NVIDIA’s GR00T N1 paper describes training its humanoid foundation model on a pyramid of data sources at different scales and levels of realism, from broad web and human video data at the base to real robot trajectory data at the top. Egocentric human video sits in the middle of that pyramid — more physically grounded than web video, more scalable to collect than real robot teleoperation.

A humanoid mid-training set occupies the layer just below real robot trajectory data: it is closer to the target embodiment than broad pretraining video, but it still leans on human capture where robot teleoperation would be slower or costlier to collect at the same coverage. Figure’s Project Go-Big, described on Figure’s official project page, is a stated commitment to scaling humanoid data collection generally — evidence that leading humanoid developers treat every layer of that pyramid, mid-training included, as worth investing in deliberately rather than skipping.

A humanoid mid-training collection pairs human contributor capture with target-humanoid capture on the same calibrated rig, so both streams share the same head-worn camera intrinsics and a single hardware clock. Two wrist cameras plus 3D hand pose keep both hands in frame with per-hand pose and visibility flags; 3D body pose and IMU add the whole-body context — stance, reach, and the transfer between locomotion and manipulation — that a humanoid mid-training set depends on more than a single-arm robot’s mid-training set would.

Coverage is set by a written spec against the target humanoid’s task list — the specific bimanual or whole-body behaviors it needs to align to — rather than collected as a generic pile of tabletop video. This is the same ten-stream, hardware-synchronized capture stack used across every Firsthand humanoid collection, applied at mid-training’s narrower, embodiment-specific scope rather than pretraining’s breadth.

Every episode ships with hand pose, contact events, action segments, instance segmentation, and 3D body pose, captured live and aligned across all ten streams under a single hardware clock. Delivery follows the same formats used across the rest of the pipeline: RLDS, LeRobot, WebDataset, or HDF5, under a perpetual, buyer-owned commercial license.

Faces, plates, and screens are irreversibly blurred at ingest, and every episode carries a documented consent chain, the same privacy and provenance handling applied to every Firsthand collection regardless of stage.

A humanoid mid-training spec has to name what a pretraining spec does not: the target humanoid’s specific head camera mounting position, field of view, arm reach, and gripper geometry, because every episode is measured against how closely it matches that geometry. It also has to name the task list the mid-training stage is meant to align to, and which side of the human-humanoid pairing already exists — humanoid teleoperation data with no matched human video, human pretraining data with no embodiment-specific alignment, or neither.

Many programs start with a Spec Pilot — a small, fast first batch that proves the spec, viewpoint match, and accept criteria before scaling volume — then run through the same seven-stage pipeline used for every Firsthand collection, with a median 48-hour first validated batch where coverage is deep and three to four weeks where it is thin and contributors need to be recruited first.

Every human contributor in a mid-training capture is enrolled through the same documented consent chain used across every Firsthand collection, and faces, plates, and screens are irreversibly blurred at ingest regardless of collection stage. Because mid-training pairs human contributor footage with target-humanoid footage on the same pipeline, both sides of the pairing carry the same provenance and consent documentation, so a buyer receives one consistent audit trail across the human and robot halves of the dataset.

Delivery is under a perpetual, buyer-owned commercial license, the same license terms used for pretraining and fine-tuning collections, so a mid-training dataset does not carry different usage rights than the rest of a program’s data.

A humanoid program with no broad bimanual or whole-body prior yet — no pretraining coverage across environments, tasks, and people — needs pretraining data first; mid-training has nothing to align if there is no broad prior underneath it. A program that already has broad pretraining coverage, or is licensing a pretrained foundation model, and needs to adapt that prior to its specific humanoid’s head camera and reach is the case mid-training data is built for.

The most common planning mistake is treating pretraining and fine-tuning as the only two stages that matter, and sizing a program as if broad human video plus a handful of task-specific humanoid demonstrations is enough on its own. EgoScale’s recipe shows the aligned mid-training step in between is what actually carries a broad prior onto a specific embodiment; without it, a fine-tuning set has to do double duty — both aligning to the humanoid and mastering the task — with far fewer hours than either job needs on its own.