Glossary
What Is Depth Data?
Last reviewed: September 2026
Short answer
Depth data is the per-pixel distance from the camera to the scene, delivered as a depth map — not a color or brightness value, but a measured distance in millimetres for every pixel. For egocentric and robotics capture, depth is derived from a calibrated stereo rig rather than a standalone depth sensor, giving a metric-millimetre map that lines up with the same-frame RGB video.
01
Why it matters
A policy or world model that only sees RGB has to infer scene geometry indirectly; a metric depth map gives it distance directly, so contact events, reach, and object geometry don’t have to be guessed from pixels alone. Depth that isn’t on the same clock as the video, or isn’t in real-world units, adds little — it has to be metric and synchronized to be useful.
02
How Firsthand handles it
Firsthand derives depth from a calibrated stereo rig rather than a standalone depth sensor, hardware-synchronized to the same clock as the video. It’s delivered as streams/depth_mm and streams/depth_conf in HDF5, observation.depth_mm in RLDS, and depth.png (16-bit, decode as uint16 millimetres) in WebDataset — see depth data for embodied AI for the full breakdown.
03
Depth fields as delivered in each format
| Format | Field | Notes |
|---|---|---|
| HDF5 | streams/depth_mm, streams/depth_conf | Metric millimetres, plus a per-pixel confidence map |
| RLDS | observation.depth_mm | Per-step key alongside observation.head_rgb |
| WebDataset | depth.png | 16-bit PNG; decode as uint16 millimetres, not as an 8-bit image |
05
Frequently asked questions
- What is depth data?
- Depth data is the per-pixel distance from the camera to the scene, delivered as a depth map in metric millimetres for every pixel — not a color value, a measured distance.
- Does Firsthand use a standalone depth sensor?
- No. Depth is derived from a calibrated stereo rig, hardware-synchronized to the same clock as the video, rather than a separate depth-sensing device. See depth data for embodied AI for the capture setup.
- What fields does depth data arrive in?
- streams/depth_mm and streams/depth_conf in HDF5, observation.depth_mm in RLDS, and depth.png in WebDataset — the PNG is 16-bit and should be decoded as uint16 millimetres, not as an 8-bit image.
- Is depth data the same as a depth sensor stream?
- Not necessarily. "Depth data" describes the output — a per-pixel distance map — regardless of how it was produced. Firsthand’s depth comes from a calibrated stereo rig rather than a standalone depth sensor.
- Why does depth need to be synchronized with the video?
- Unsynchronized depth doesn’t line up with the RGB frame it’s meant to describe, so a model can’t reliably fuse the two. Depth only adds geometric grounding when it shares the same clock as the video it accompanies.