Why Stereo Video Matters for Robot Pretraining Data

A pretraining corpus is meant to teach a policy broad visuomotor priors before it ever touches a specific robot. Depth is part of that prior: knowing how far an object actually is, and how a hand approaches it in three dimensions, matters most for contact-rich manipulation and precise grasping. Monocular RGB alone gives the model color and texture but no built-in geometry — depth either has to be inferred indirectly or skipped. A calibrated stereo pair solves this at capture time, recovering depth from disparity rather than asking a downstream model to guess it.

That distinction compounds at pretraining scale. A single monocular clip missing depth is a minor gap; a multi-hundred-hour pretraining corpus missing depth across every episode is a structural limitation on what the resulting prior can represent, however diverse the tasks and scenes are otherwise.

The choice is also not purely academic. Teams sourcing pretraining data for a manipulation-heavy policy are effectively deciding, at spec time, whether depth is a first-class training signal or something the downstream model has to approximate from texture, occlusion, and motion cues alone. A written spec that names stereo capture up front avoids having to re-collect or bolt on a depth sensor later, once the gap becomes visible in evaluation.

Stereo capture records the same scene through two synchronized, calibrated cameras spaced at roughly a human interpupillary distance, so every frame pair carries recoverable depth from geometry alone. Monocular capture is a single camera stream — it can still carry hand pose, IMU, and audio, but it has no second viewpoint to triangulate against, so depth has to come from somewhere else if the task needs it.

Two cameras at a fixed baseline — close to human interpupillary distance — record the same scene, and the disparity between the left and right image is triangulated into dense, pixel-aligned depth from geometry alone. Because it is solved from geometry rather than an emitted signal, stereo depth works outdoors where active depth sensors struggle. Every episode is bracketed by an opening and closing calibration solve and ships only if RMS reprojection stays under 0.28 px, and depth hardware-triggers onto the same PTP clock domain as RGB, IMU, and hand pose.

Hand pose and wrist-level action labels are more reliable when they are anchored to real 3D geometry instead of estimated from a single 2D view. With a calibrated stereo pair, hand keypoints and wrist trajectories can be triangulated against the same disparity that produces the depth stream, so the action label and the depth are geometrically consistent with each other rather than two independently estimated signals that might disagree. That consistency matters most for contact-rich manipulation, where the exact moment and geometry of contact is the signal a policy needs to learn.

This site’s action-labels glossary entry describes the range of label types — hand pose, action segments, contact events, latent actions, and inverse-dynamics pseudo-labels — that pretraining corpora use; see /glossary/action-labels for the full breakdown.

The published egocentric corpora referenced on this site each document their own action representation. NVIDIA EgoScale, described in its paper on arXiv and on NVIDIA’s GEAR lab research page, represents actions as wrist-level 6-DoF motion paired with articulated hand pose, produced from off-the-shelf SLAM and hand-pose estimation applied after capture rather than measured on a synchronized stereo rig. Generalist AI’s GEN-0, announced in its official November 2025 blog post, is trained on more than 270,000 hours of data with a stated emphasis on data quality and diversity over raw hour count, but its announcement does not detail camera geometry. EgoVerse (arXiv 2604.07607) and Egocentric-10K (its official Hugging Face dataset card) are documented by hour count and episode count — roughly 1,362 hours across about 80,000 episodes for EgoVerse, and roughly 10,000 hours for Egocentric-10K — without this site claiming a specific camera modality for either beyond what their own pages state.

This site does not assert that any of these published corpora are monocular or stereo beyond what their own official pages document. Verify camera geometry against the linked source before relying on it for a specific pretraining plan.

The practical implication is that a team evaluating whether a public corpus already covers a stereo-depth requirement cannot answer that question from hour counts and scaling claims alone — those numbers describe scale, not sensor geometry. EgoScale’s wrist-level action representation, for instance, was derived after capture through SLAM and hand-pose estimation applied to recorded footage, which is a different pipeline than triangulating depth from a synchronized stereo pair at the moment of capture. Both approaches produce usable action labels, but only one produces a depth stream that is geometrically consistent with those labels by construction.

Mid-training bridges a human-video pretraining prior to a specific robot embodiment, and that bridge is easier to build when the human-side geometry already matches how the robot itself is instrumented. If a target robot carries its own calibrated multi-camera or depth-capable rig, pretraining data that already ships metric depth and 3D hand pose reduces the representational gap the mid-training stage has to close, compared with a monocular prior that never carried depth at all. Where the target robot has no depth sensing of its own, or the task is simpler navigation or classification, that advantage narrows and monocular pretraining data is the more efficient choice.

This is the same logic GR00T N1’s data pyramid uses when it stitches heterogeneous sources into one training curriculum: what matters for alignment is not that every layer is captured identically, but that each layer’s geometry and action representation are documented well enough to be mapped onto the target robot’s own instrumentation during mid-training. A stereo pretraining layer that ships its calibration block alongside depth and hand pose gives that mapping step more to work with than a monocular layer that never recorded geometry in the first place.

A stereo pretraining spec ships the rectified or raw left/right pair, the depth_mm and depth_conf streams described above, hand pose and wrist-action labels triangulated against the same geometry, and the calibration block needed to re-derive depth independently. Every stream hardware-triggers onto one PTP clock domain, so RGB, depth, IMU, and pose share a timebase rather than being merely co-recorded — the measured residual across delivered episodes runs a median of 1.12 ms and a p99 of 1.94 ms, with a 2.0 ms reject ceiling.

• Left/right RGB pair, raw or rectified on request
• depth_mm and depth_conf, metric millimetres with per-pixel confidence
• Hand pose and wrist-level action labels, geometrically consistent with the depth stream
• Per-episode calibration block: intrinsics, extrinsics, and stereo baseline
• Delivered in RLDS, HDF5, or zarr

Stereo pretraining data is collected against a written spec rather than pulled from a fixed catalog, so reach, task and scene diversity, and timeline are quoted against what a policy actually needs. A first validated batch ships in a median of 48 hours where coverage is deep, whether the spec calls for a stereo rig, a monocular one, or both on the same episode.

The spec itself is where the stereo-vs-monocular decision gets made concrete: naming the target task, the environments it needs to generalize across, and whether depth is a training signal the downstream model consumes directly settles the rig choice before capture starts, rather than after a first batch turns out to be missing a stream the policy needed.