Stereo Cameras vs RGB-D Sensors for Robotics Data Collection

A stereo camera pair is two ordinary RGB cameras mounted at a fixed, calibrated distance apart. Depth is not measured directly — it is computed after the fact, by finding the same point in both images and triangulating how far apart it appears (its disparity) to work out how far away it is. Nothing is emitted into the scene; the two color images alone contain enough geometry to recover depth.

An RGB-D sensor is a single unit that pairs a color camera with an active depth-sensing element, most commonly a structured-light projector or a time-of-flight emitter. It emits a signal — a light pattern or a pulsed light source — into the scene and measures how that signal is distorted or how long it takes to return, producing a depth map directly from the sensor rather than by computing it from two images.

Both approaches end up producing the same kind of output for a training pipeline — a per-pixel depth map alongside an RGB image — but they get there through fundamentally different physics, and that difference has real consequences for egocentric robotics capture.

Stereo depth is a computation, not a sensing modality on its own. Two calibrated cameras record the same scene from slightly different viewpoints, and software matches corresponding points between the left and right images to compute disparity, then converts that disparity into a distance using the measured baseline, intrinsics, and extrinsics. Because it works entirely from ordinary color images, stereo depth degrades gracefully in bright sunlight and works at any range the two cameras can both resolve.

RGB-D depth is a direct measurement from a dedicated depth-sensing element. A structured-light sensor projects a known pattern and reads how that pattern deforms across the scene; a time-of-flight sensor emits pulsed or modulated light and measures the delay or phase shift of the return. Both approaches depend on the emitted signal reaching the scene and coming back cleanly, which is why they are generally more sensitive to strong ambient light, highly reflective or absorptive surfaces, and effective range than a passive camera pair.

Egocentric capture is worn on a person’s head and wrists while they perform real tasks, often outdoors or under whatever lighting a real environment happens to have. A calibrated stereo pair is two passive cameras — it adds no active emitter to fight sunlight, and it fits into a head- or wrist-mounted rig at roughly the size of a pair of ordinary cameras plus a known baseline between them. That is why Firsthand derives depth from a calibrated stereo rig rather than a standalone depth sensor: it keeps the rig passive and keeps depth accurate outdoors and indoors alike.

RGB-D sensors are a strong fit where the capture is stationary, indoor, and at a controlled range — a fixed tabletop rig scanning an object, for example — where an active sensor’s direct depth measurement can simplify the pipeline. Worn on a moving head or wrist, an active emitter has to contend with ambient light, motion, and whatever surfaces are in front of it at a given instant, conditions a passive stereo pair does not depend on in the same way.

A stereo rig needs its own calibration before disparity means anything: the intrinsics of each lens, the extrinsics between the two cameras, and the baseline distance between them all have to be measured, and drift in any of those numbers corrupts depth in a way that grows with distance from the cameras. Firsthand measures the baseline, intrinsics, and extrinsics every session and brackets each episode with an opening and closing calibration solve, shipping only if RMS reprojection stays under 0.28 px, and ships those calibration values with the episode so rectification is exact rather than assumed.

An RGB-D sensor ships its own factory calibration between the color camera and the depth-sensing element, so a buyer does not solve stereo geometry themselves — but that calibration is fixed by the manufacturer rather than measured per session, and the two data streams (color and depth) still have to be aligned and synchronized to the rest of the capture rig.

Sync matters regardless of which approach produces the depth. In Firsthand’s rig, depth, RGB, IMU, and hand pose all hardware-trigger onto the same PTP clock domain, so a depth frame and an RGB frame captured at the same instant carry matching timestamps rather than being aligned after the fact in software. Whatever the depth source, unsynchronized depth does not line up with the RGB frame it is meant to describe, and a policy cannot reliably fuse the two.

A calibrated stereo capture delivers a per-pixel depth map in metric millimetres, not a color or brightness value, alongside a confidence channel that flags low-confidence pixels. Firsthand ships depth as streams/depth_mm and streams/depth_conf in HDF5 and zarr, as an observation.depth_mm step key in RLDS, and as a 16-bit depth.png (decode as uint16 millimetres, not as an 8-bit image) in WebDataset shards. Raw stereo frames ship with the session’s calibration so a buyer can re-rectify independently, and rectified pairs plus disparity can be added to the delivery on request.

Depth is not embedded in an mp4 video file — it ships as its own array or file per episode, so a policy pulling most of its streams from a LeRobot mp4-based delivery should pull depth from the HDF5 or RLDS export instead.

A single RGB-D unit gives a direct depth reading without solving stereo geometry, which is a real advantage for a stationary or near-stationary rig scanning a bounded working volume at a controlled range and lighting level — a fixed capture station for object scanning, for example, rather than a rig moving through a real task. Programs already built around an existing RGB-D-based pipeline may also find continuity with that sensor’s output more valuable than switching capture modalities mid-program.

The two approaches are not mutually exclusive across a program: some teams capture both a calibrated stereo signal and a fixed RGB-D reference and synchronize them onto one clock, because they solve the depth problem through different physics and can validate each other.

Start from where the capture happens and how it moves. Worn, head- or wrist-mounted capture across real environments — indoors and outdoors, under whatever lighting the task actually has ��������������� favors calibrated stereo, because it is passive and does not depend on an emitted signal returning cleanly. A fixed, indoor rig at a controlled range, where minimizing calibration effort matters more than mobility, is a reasonable case for RGB-D.

From there, the decision should follow what the policy actually needs from depth: whether it has to be synchronized on the same hardware clock as RGB, IMU, and hand pose; what format the delivery needs to land in; and whether the task is contact-rich manipulation, where dense, pixel-aligned depth alongside 3D hand pose matters most, or a simpler task where depth may not be needed at all. Either way, a custom collection is scoped against a written skill spec rather than pulled from a fixed catalog, so the capture modality is part of that spec conversation, not a default.