Guide
Depth Data for Embodied AI
Short answer
Depth data gives a policy the metric distance to every pixel, not just color. Firsthand recovers it from a calibrated stereo pair rather than a separate active sensor, so it stays accurate outdoors and indoors alike. It ships as a synchronized depth_mm stream in millimetres, aligned to RGB and hand pose on the same hardware clock, in HDF5, zarr, RLDS, or WebDataset.
Last reviewed:
What is depth data, and why does embodied AI need it?
Depth data records the distance from the camera to every point in a scene, on top of the color image an RGB stream already carries. A manipulation policy trained on RGB alone has to infer geometry indirectly; a policy trained with metric depth can learn how far an object actually is and how a hand approaches it in 3D, which matters most for contact-rich manipulation and precise grasping.
How is depth captured for a Firsthand dataset?
Depth comes from a calibrated stereo pair rather than a separate active depth sensor. Two cameras at a fixed, human-eye baseline record the same scene, and the disparity between the left and right image is triangulated into dense, pixel-aligned depth from geometry alone. Because it is solved from geometry rather than an emitted signal, stereo depth works outdoors where active depth sensors struggle.
- Two synchronized, calibrated cameras at a fixed baseline recover depth from disparity
- Every episode is bracketed by an opening and closing calibration solve, shipping only if RMS reprojection stays under 0.28 px
- Depth hardware-triggers onto the same PTP clock domain as RGB, IMU, and hand pose
What does a delivered depth stream actually look like?
Depth ships as its own array or file per episode, in metric millimetres rather than a relative or normalized scale, alongside a confidence channel that flags low-confidence pixels.
| Field | What it carries |
|---|---|
| streams/depth_mm | Metric depth, uint16, in millimetres — HDF5 delivery |
| streams/depth_conf | Per-pixel confidence channel alongside depth_mm — HDF5 delivery |
| observation.depth_mm | Step-level depth key — RLDS delivery |
| depth.png | 16-bit depth, decode as uint16 millimetres — WebDataset shard key |
| Calibration | Intrinsics, extrinsics, and stereo baseline, shipped per episode |
| Sync | Depth, RGB, IMU, and hand pose on one hardware-triggered clock |
Depth is a synchronized stream, not a one-off render — it carries the same calibration and clock guarantees as the rest of the episode.
Depth is not embedded in an mp4 video file — it ships as its own array or file, so a policy pulling depth from a LeRobot mp4-based delivery should pull it from the HDF5 export instead.
When does a policy actually need depth?
Depth matters most for 3D hand pose, object pose, and contact-rich manipulation, where a policy needs to know how far a hand or object actually is rather than just where it appears in the image. Simpler tasks, such as navigation, activity recognition, or 2D classification, are often fine on a monocular RGB stream with no depth at all.
FAQ
Depth Data for Embodied AI, answered.
01Does Firsthand capture depth with a stereo rig or a separate depth sensor?
A calibrated stereo pair, not a separate active depth sensor. Two cameras at a fixed baseline recover dense depth from disparity, which is why it holds up outdoors, where active sensors struggle.
02What units does the depth stream use?
Metric millimetres, delivered as depth_mm alongside a depth_conf confidence channel, rather than a relative or normalized depth scale.
03What formats does depth ship in?
HDF5 and zarr carry depth natively as its own array. RLDS carries it as an observation.depth_mm step key. WebDataset carries it as a 16-bit depth.png per sample. It is not embedded in an mp4 file, so pull it from the HDF5 export if a delivery is otherwise mp4-based.
04Is depth synchronized with the RGB and hand-pose streams?
Yes. Depth, RGB, IMU, and hand pose all hardware-trigger onto the same PTP clock domain, so a depth frame and an RGB frame captured at the same instant carry matching timestamps.
05Do all tasks need a depth stream?
No. Depth matters most for 3D hand pose, object pose, and contact-rich manipulation. Simpler navigation or classification tasks are often fine on monocular RGB alone.
Sources
Where is this documented?
- RLDS — Reinforcement Learning Datasets format (opens in a new tab)The episode format used for RLDS delivery, including the observation.depth_mm step key. github.com
- WebDataset (opens in a new tab)Streaming tar-based format used for depth.png shard delivery. github.com
Check it against the sample pack.
40 episodes across 4 environments, delivered in the exact schema these guides describe.