How to Set Up a Stereo Camera Rig for Robotics Data Collection

A stereo camera rig is two cameras mounted at a fixed, known distance from each other so the disparity between their left and right images can be turned into geometric depth for every pixel. For egocentric robotics data, that pair is head-mounted so it captures the scene from roughly the operator’s own viewpoint — the vantage a humanoid or head-mounted robot camera actually sees from — rather than from a third-person angle looking in.

On its own, a stereo pair only gives you two slightly offset video streams. What makes it useful for robot learning is everything around it: the cameras have to be individually calibrated, the pair has to be jointly calibrated against each other, and the whole rig has to share a clock with whatever else is capturing the same episode — wrist cameras, a depth sensor, an IMU. This guide walks through each of those pieces in the order you’d set them up.

This page covers rig setup end to end. For the calibration procedure specifically, see camera calibration procedure for a multi-camera egocentric rig. For hardware and QA context, see /capture.

A global-shutter camera exposes every pixel in the frame at the same instant. A rolling-shutter camera instead exposes row by row over a short interval, so fast motion during that interval skews and warps the image — exactly the kind of artifact that shows up during a reach-and-grasp motion, which is also the motion you most want a policy to learn from.

That matters more for a stereo rig than for a single camera, because depth is recovered by matching corresponding points between the left and right images. If one or both cameras have rolling-shutter skew, the left and right images no longer describe the same instant in time in the same way row by row, and the disparity match — and the depth derived from it — degrades exactly during the fast motion that a manipulation policy cares about most. A global shutter on both cameras keeps the whole frame geometrically consistent with the depth and pose streams captured alongside it.

For egocentric robotics data, the stereo pair mounts on a head-worn platform so the viewpoint matches what a humanoid or head-mounted robot camera would actually see — not a third-person angle, and not a single monocular lens. The two lenses are set at a baseline close to human interpupillary distance, so the geometry a policy learns from mirrors the viewpoint it will operate from, rather than a wider or narrower baseline chosen for some other reason.

The stereo pair is usually only one piece of the rig. Firsthand’s head-mounted platform pairs the stereo cameras with two wrist-mounted cameras for close-range manipulation detail, an active-stereo depth sensor, and IMUs at the head and each wrist — all sharing the same hardware trigger and clock. Where you place the wrist cameras and depth sensor relative to the head-mounted stereo pair matters less than making sure every sensor’s pose relative to the others is solved during calibration, not assumed from the mechanical drawing.

The exact numeric stereo baseline is a calibration value measured and shipped per session, not a fixed spec you can read off a data sheet — see stereo baseline for what it is and why it’s measured rather than assumed.

Calibration for a stereo rig solves the same intrinsics every single camera needs — focal length, principal point, and lens distortion — plus the extrinsics that describe how the two stereo cameras (and any other camera on the rig) are positioned and oriented relative to each other. Firsthand runs this as a full opening-and-closing procedure rather than a one-time factory calibration:

Bracketing every session with an opening and a closing solve turns rig drift — a worn head mount flexing over a session — from an assumption into a measured number that ships with the episode as calib_drift_mm. An episode is only accepted if RMS reprojection stays under 0.28 px and the measured drift is within tolerance; otherwise it goes to the reject log rather than shipping with unverified geometry.

Calibration solves where each sensor is; synchronization solves when each sensor fired. Both matter for a stereo rig, because a stereo match computed from frames captured at slightly different instants is only as good as its worst-aligned pixel. Firsthand runs every sensor on the rig — the stereo pair, both wrist cameras, the depth sensor, and the head- and wrist-mounted IMUs — in a single PTP (Precision Time Protocol) clock domain, with a PTP grandmaster in the belt pack and a hardware trigger line to every sensor, rather than relying on software timestamps from each device’s own clock.

The IMUs run at 200 Hz — faster than the camera frame rate — from three 6-DoF (accelerometer plus gyroscope) BMI088 sensors, one head-mounted and one on each wrist. Because the IMU samples more often than the cameras capture frames, an episode carries more IMU samples than video frames for the same span of time; that is expected, and consuming the two streams together means resampling or windowing the IMU data to the frame you need rather than assuming a 1:1 sample-to-frame mapping.

Every delivered episode carries per-stream int64 nanosecond timestamps in the PTP domain, so the sync error between any two streams can be recomputed from the data itself rather than taken on trust — see /capture for the reproducible sync-measurement method and code.

Depth from a calibrated stereo rig comes from matching corresponding points between the left and right images and converting the disparity between them into distance, using the calibrated baseline, intrinsics, and extrinsics rather than a single-camera estimate. That is a different physical measurement from monocular depth estimation, which infers depth from a single image’s learned cues and carries no calibrated geometric guarantee.

Firsthand registers depth to the head-camera frame and ships it alongside the RGB and pose streams. Raw stereo frames ship with the session’s calibration for custom re-rectification, or rectified pairs plus disparity can be added to the delivery.

A stereo rig can pass calibration in the morning and still produce unusable data by the afternoon if something drifts, is occluded, or falls outside the sync ceiling mid-session. Firsthand applies rejection thresholds automatically at ingest — no human decides a drifted session is “probably fine” — against the same criteria worth checking on any rig:

Everything the rig captures — stereo RGB, wrist RGB, depth, and IMU — is delivered as separate, timestamped streams rather than pre-merged into a single track, so a training pipeline can pull exactly the modalities it needs. The stream naming stays consistent across HDF5, RLDS, and WebDataset delivery; only the container and per-sample convention change.

• streams/head_rgb, streams/wrist_l, streams/wrist_r — RGB video from the head-mounted stereo pair and each wrist camera
• streams/depth_mm, streams/depth_conf — metric depth and per-pixel confidence, registered to the head-camera frame
• streams/imu_head, streams/imu_wrist_l, streams/imu_wrist_r — 6-DoF IMU data at 200 Hz, one array per sensor
• *_timestamp_ns — int64 nanosecond timestamps in the shared PTP domain, present per stream

See /formats/episode-schema and /formats for the full field-by-field breakdown across HDF5, RLDS, LeRobot-style step keys, and WebDataset shards.