How to Collect Aligned Human-Robot Mid-Training Data

Aligned human-robot mid-training data is the smaller, embodiment-matched dataset collected in the middle stage of a three-stage robot foundation model training recipe. Broad pretraining comes first, drawing on large-scale human egocentric video or mixed data collected without any single robot in mind. Mid-training comes next: a much smaller, tightly aligned slice of human-plus-robot data that adapts the pretrained representation to one target robot’s embodiment — matched viewpoint, calibration, and tasks. Task-specific fine-tuning (post-training) narrows further still, to the exact deployment task. This same three-stage structure is documented in full in pretraining vs mid-training vs post-training robot data.

The word "aligned" is doing real work in that definition. A mid-training set is not just a smaller pretraining set; it is purpose-built to match one robot’s camera geometry, reach, and task set as closely as possible, so that the comparison between a human episode and a robot episode is apples-to-apples rather than two unrelated recordings. That alignment is what this page walks through collecting, step by step.

This page describes the collection methodology. For a commissioned service that runs this process against a project’s mid-training spec, see aligned human-robot mid-training data.

A pretrained representation teaches general visual and manipulation priors from human video at scale, but it is not aligned to any one robot’s camera geometry, field of view, or reach. Adding more pretraining hours widens the prior; it does not narrow that gap. Mid-training closes the gap a different way — by recording from a viewpoint and calibration that match the target robot as closely as possible, on a much smaller dataset built specifically for that purpose.

EgoScale’s published recipe is the clearest public illustration of this trade-off. Its pretraining corpus runs to 20,854 hours of human egocentric video; its mid-training stage pairs roughly 50 hours of human video with about 4 hours of robot data across 344 tabletop tasks. That mid-training slice is a small fraction of the pretraining corpus, and the published result is that the aligned mid-training stage — not the pretraining volume alone — is what carries the prior onto a specific robot, reported as a 54% success improvement on a 22-DoF hand (arXiv 2602.16710; NVIDIA GEAR Lab project page). The lesson generalizes beyond that one number: the value of mid-training data comes from how closely it matches the target robot, not from its size.

This is the same lesson GEN-0’s official write-up draws about pretraining diversity mattering as much as volume, applied one stage later: at the mid-training stage, alignment to a specific embodiment matters more than adding hours (GEN-0 blog post).

Matching viewpoint means the human contributor’s capture rig is positioned, mounted, and calibrated to resemble the target robot’s cameras as closely as possible, rather than recorded from whatever angle is convenient. If the robot has head-mounted or chest-mounted cameras at a certain height and field of view, the human capture setup is matched to that height and field of view, not to a generic egocentric framing. The same logic applies to wrist-mounted cameras: if the robot’s manipulation view comes from a wrist camera, the human contributor’s capture setup includes a comparable wrist-level view.

Matching the viewpoint is only useful if the resulting camera parameters are also known precisely, which is why calibration is inseparable from viewpoint matching rather than a separate concern. A camera positioned at the right height but with uncalibrated intrinsics still leaves the mid-training stage’s core promise — that the human and robot streams describe the same scene geometry — unmet.

A mid-training spec should name the target robot’s camera mounting points and fields of view up front, the same way a pretraining spec names task categories and scene types, so viewpoint matching is a checkable requirement rather than something assessed informally after footage already exists.

Mid-training data pairs a human contributor and the target robot performing the same or closely related tasks, on the same capture and calibration setup, so the two streams stay directly comparable episode for episode. The task list for a mid-training collection is therefore drawn from the robot’s actual deployment tasks — or a closely related superset of them — rather than a broad, exploratory task list the way a pretraining spec might use.

EgoScale’s published recipe illustrates this at scale: its mid-training stage covers 344 tabletop tasks, a scoped and specific task set rather than an open-ended one, paired across human and robot capture on the same pipeline. The task list is small enough to be enumerable and checkable against a spec, which is exactly the property a pretraining task list often does not need and a mid-training task list always does.

In practice this means the task list is one of the first things fixed in a mid-training spec, before scheduling any capture sessions. A spec that leaves the task list vague — "general manipulation," for instance, instead of a named list of pick, place, and tool-use tasks matched to the robot’s deployment envelope — makes it impossible to check after the fact whether the collected data actually overlaps with what the robot needs to learn.

Every camera on a mid-training rig gets its own intrinsic solve — focal length, optical center, and lens distortion coefficients (fx, fy, cx, cy, k1, k2, p1, p2, k3) — plus an extrinsic solve expressing where it sits in 3D space relative to the others, as a 4×4 SE(3) transform with the head-camera frame as identity. That is the same intrinsic-plus-extrinsic calibration procedure used for any egocentric rig, described in full in calibrating head and wrist cameras together; mid-training data does not get a different calibration procedure, it just gets measured against a specific robot’s intrinsics rather than a generic one.

Hand-pose data is recorded in metres in the head-camera frame, which only means what it claims if intrinsics and extrinsics are measured accurately — so each rig session is checked against a reprojection-error threshold (rms_reproj_px < 0.28) before its frame-level labels are considered trustworthy. That same gate applies whether the session is a pretraining capture or a mid-training capture, because both depend on the same calibration discipline underneath them.

Time sync matters even more directly for mid-training than for pretraining, because mid-training explicitly pairs a human episode with a robot episode and expects the two to be comparable frame for frame. Every stream — video, depth, hand pose, and action_segments — is hardware-triggered onto the same PTP clock domain, so a query for what the human contributor was doing at a given timestamp and what the robot is doing at the matched timestamp is a direct lookup rather than a best-effort reconstruction after the fact. Cross-stream sync error, the residual timing offset between simultaneously captured streams, is measured the same way regardless of whether the capture is human or robot, because both are only as good as the clock alignment underneath them.

Mid-training data carries the same label types as any egocentric capture — hand pose and action segments — recorded on the same PTP clock domain as the video and depth streams. Action segments are delivered as action_segments records with a {start_ns, end_ns, verb, noun} structure and nanosecond timestamps, so a segment boundary lines up with the exact frame and hand-pose sample where a sub-action actually starts or ends.

What is different at the mid-training stage is not the label schema but how the labels are used: because a human episode and a robot episode on the same or closely related task are meant to be directly comparable, having both episodes labeled with the same verb-noun vocabulary and the same clock domain is what makes that comparison possible at all. A human "grasp mug" segment and a robot "grasp mug" segment are only comparable if both are timestamped against the same underlying structure.

Contact events and frame-level hand-pose fields follow the same logic: they are recorded the same way regardless of whether the contributor is a human or the target robot, so that the frame-level detail available for one stream is also available for the other.

Mid-training data goes through the same underlying quality checks as any egocentric capture — sync, calibration, and consent — before an episode counts as a validated hour. Cross-stream sync error is checked against the same ceiling; hand-pose geometry is checked against the per-session reprojection-error threshold; and consent is documented for every contributor the same way it is documented for any other collection.

On top of those universal checks, a mid-training QA pass also checks alignment itself: does the human episode’s viewpoint actually resemble the robot’s camera geometry, and does the task list in the delivered data actually match the task list named in the spec. These are coverage checks in the same spirit as the diversity-coverage checks run on a pretraining dataset, just measured against a narrower target — one robot’s embodiment and task list, rather than broad scene, object, and participant diversity.

The pilot-before-scaling discipline used for pretraining collection applies here as well: a small aligned batch is captured and checked against the spec — sync, calibration, task coverage, and viewpoint match — before a full mid-training collection scales up, so a mismatch in rig setup or task list is caught while it is still cheap to fix.

EgoScale is the clearest public reference point for mid-training scale: its recipe pairs roughly 50 hours of human video with about 4 hours of robot data, across 344 tabletop tasks (arXiv 2602.16710; NVIDIA GEAR Lab project page). That is dozens of hours, not thousands — a small fraction of EgoScale’s own 20,854-hour pretraining corpus.

The ratio between the two stages is the useful planning takeaway, not a specific hour target to copy for every program: pretraining scale is measured in tens of thousands of hours because it needs breadth across many tasks and settings; mid-training scale is measured in dozens of hours because it needs precision on one embodiment. A mid-training program sized to a different robot or task set will land on different hour counts, but the underlying principle — small and tightly aligned beats large and loosely matched — is the one EgoScale’s published numbers demonstrate.

Because a mid-training set is scoped tightly to one robot’s viewpoint, calibration, and task list, it is a poor fit for a static, off-the-shelf dataset — the alignment requirement is specific to the buyer’s own robot, not something a pre-existing corpus is likely to already match. On-demand collection to a written spec is the natural fit: the spec names the target robot’s camera mounting points, fields of view, and task list, and a custom collection is run against that spec rather than filtered out of existing footage after the fact.

Firsthand captures matched human-and-robot mid-training data — synchronized on a single hardware clock, with calibrated intrinsics and hand pose or action-segment labels — collected fast, on demand, and to a written spec, then delivered in RLDS, LeRobot format, or WebDataset with a buyer-owned commercial license. Because mid-training data is measured against one specific robot, the spec centers on matching that robot’s viewpoint and reach rather than maximizing hours, and a human contributor and the target robot typically perform the same or closely related tasks on the same capture and calibration setup so the two streams stay directly comparable frame for frame.

The same pilot-before-scaling discipline that applies to pretraining collection applies here: a small aligned batch is captured and checked against the spec before a full collection scales up, so viewpoint-match or task-coverage problems are caught while the cost of fixing them is still small.

The table below summarizes the steps this page walks through, as a checklist for planning a mid-training collection program.