Guide
Stereo vs Monocular Video for Robot Learning
Short answer
Stereo video uses two calibrated cameras at a human eye baseline to recover dense, pixel-aligned depth directly from geometry; monocular uses one camera and either skips depth or infers it separately. Stereo suits 3D manipulation and contact-rich tasks; monocular suits simpler navigation or classification work where a single RGB stream is enough. Firsthand collects both on demand.
Last reviewed:
What is the difference between stereo and monocular video for robot learning?
Stereo capture records the same scene through two calibrated cameras spaced like human eyes, so every frame pair carries recoverable depth from geometry alone. Monocular capture is a single camera stream — it can still carry hand pose, IMU, and audio, but it has no second viewpoint to triangulate against, so depth has to come from somewhere else if the task needs it.
- Stereo: two synchronized, calibrated cameras at a fixed baseline, plus whatever else the rig carries
- Monocular: one camera stream, plus whatever else the rig carries
- Both are hardware-triggered onto the same clock and collected on demand to a written spec
Why does stereo recover depth that monocular cannot?
A calibrated stereo pair turns the disparity between the left and right image into depth, pixel by pixel, from geometry that is measured every session. A single camera has no second view to triangulate against, so a monocular stream either ships without depth or needs a separate sensor added to the rig.
| Property | Stereo | Monocular |
|---|---|---|
| Depth source | Dense disparity from a calibrated left/right pair | Not captured directly — add a depth sensor to the rig if needed |
| Calibration | Intrinsics, extrinsics, and stereo baseline, shipped per episode | Intrinsics only; no baseline to solve |
| Geometry | Human interpupillary baseline, matching head-mounted viewpoints | Single viewpoint, no binocular disparity |
| Best for | 3D hand pose, object pose, contact-rich manipulation | Navigation, activity recognition, simpler 2D tasks |
| Formats | RLDS · HDF5 · zarr | RLDS · LeRobot · WebDataset |
| Sync | Left/right pair plus depth, IMU, hand pose on one hardware clock | RGB stream plus any added sensors on one hardware clock |
Both modes are collected on demand to a written spec, with the same synchronization and calibration discipline.
When is monocular egocentric capture enough?
Monocular is the simpler, cheaper rig, and it is enough whenever the policy does not need metric depth from the camera itself — navigation, activity or scene recognition, or tasks where a paired depth sensor already covers the geometry. It is also the right default when hand pose, IMU, and audio are the streams that matter and stereo geometry would add cost without adding signal.
How does Firsthand capture stereo or monocular video on demand?
Both modes run the same discipline. Every sensor hardware-triggers onto one PTP clock domain, so RGB, depth, IMU, and pose share a timebase rather than being merely co-recorded — the measured residual across delivered episodes runs a median of 1.12 ms and a p99 of 1.94 ms, with a 2.0 ms reject ceiling. Every episode is also bracketed by an opening and closing calibration solve, and ships only if RMS reprojection stays under 0.28 px.
The full sync and calibration procedures, with the numbers verified against real episode data, are in "How to measure cross-stream sync error" and "Camera calibration procedure" below.
How fast can a custom stereo or monocular batch ship?
Either mode is collected against a written spec rather than pulled from a fixed catalog, so reach, timeline, and price are quoted against what your policy actually needs. A first validated batch ships in a median of 48 hours where coverage is deep, whether the spec calls for a stereo rig, a monocular one, or both on the same episode.
FAQ
Stereo vs Monocular Video for Robot Learning, answered.
01Can I combine stereo and monocular cameras in the same capture rig?
Yes. A head-mounted stereo pair, wrist cameras, and a third-person reference camera can all trigger on the same hardware clock in one episode, so you are not locked into one mode for the whole rig.
02Does monocular capture ship without depth?
By default, yes — a single RGB stream has no built-in depth. If your spec needs depth alongside a monocular stream, a depth sensor is added to the rig and synchronized on the same clock as the RGB and IMU.
03Is stereo depth more accurate than a depth sensor?
They solve the problem differently rather than one simply beating the other: stereo recovers depth from calibrated geometry and works outdoors where active depth sensors struggle, so many programs capture both and sync them on one clock.
04What formats does stereo vs monocular data ship in?
Both ship in RLDS, LeRobot, and WebDataset, with stereo additionally available in HDF5 and zarr to carry the rectified pair and disparity alongside the RGB and depth streams.
05Can I request rectified stereo pairs instead of raw?
Yes. Raw frames ship with calibration so you can re-rectify yourself, and rectified pairs plus disparity can be added to the delivery on request.
06How is sync verified across stereo left and right frames?
Every sensor, including both stereo cameras, hardware-triggers onto one PTP clock domain; the measured residual across delivered episodes runs a median of 1.12 ms and a p99 of 1.94 ms, with a 2.0 ms reject ceiling.
Sources
Where is this documented?
- RLDS — Reinforcement Learning Datasets format (opens in a new tab)The episode format used for RLDS delivery. github.com
- LeRobot (opens in a new tab)Hugging Face's robot-learning framework and dataset format. github.com
- WebDataset (opens in a new tab)Streaming tar-based format used for large-scale delivery. github.com
- Rerun (opens in a new tab)Visualization tool used to inspect multi-stream recordings, including stereo and depth. www.rerun.io
Check it against the sample pack.
40 episodes across 4 environments, delivered in the exact schema these guides describe.