What Is Stereo Disparity?

A model that sees only RGB has to guess distance from cues like size and occlusion. Disparity gives it a measurement instead, because the shift between two views follows from geometry. That only holds when the geometry is known: disparity is a pixel count until the baseline and focal length behind it are measured, so stereo depth is only as trustworthy as its calibration.

Firsthand derives depth from a head-mounted, calibrated stereo rig at a human interpupillary baseline, not from a standalone depth sensor. Intrinsics, extrinsics and the baseline are measured every session and shipped with the episode, and depth arrives in metric millimetres on the same clock as the video. The site states no baseline length, depth range, depth accuracy or matching algorithm.

Two cameras mounted side by side see the same scene from slightly different positions. A point on a cup, a hand or a table edge therefore lands at slightly different horizontal positions in the left and right images. That difference, measured in pixels, is the stereo disparity of that point.

The textbook setup uses a rectified pair, where the images are transformed so matching points sit on the same row and the shift is purely horizontal. Nearby points have a large disparity; distant points have a small one, approaching zero. Disparity is measured in pixels, not distance, so it needs calibration before it means anything physical.

The standard relation is depth = focal length × baseline / disparity. The OpenCV stereo depth tutorial writes the same triangle geometry as disparity = x − x′ = B·f / Z, where B is the distance between the two cameras, f is the focal length and Z is the depth of the point. Its summary is that the depth of a point is inversely proportional to the difference between its image positions.

Two consequences follow. You need B and f: without a measured baseline and calibrated focal length, disparity cannot become distance. And because depth varies with the inverse of disparity, a one-pixel matching error matters more for distant points, where disparity is small.

A disparity map is an image in which each pixel holds the disparity found for it, produced by searching along the row of the other image for the best-matching patch. The OpenCV tutorial shows a block matcher (StereoBM) producing one from a left and right image.

That tutorial notes the first result is noisy and needs tuning: texture thresholds filter areas too featureless to match, speckle filters clean artifacts at object boundaries, and a number-of-disparities setting widens the range of visible depths. A disparity map is therefore an estimate, not ground truth. Converting each valid pixel through the depth relation gives a depth map; unmatched pixels have no reliable depth.

Contact-rich manipulation depends on small distances, such as hand-to-handle gap or grasp clearance. The site’s stereo guidance states that a pair gives geometric depth directly from disparity, where monocular video leaves scale to be inferred from cues like size and occlusion.

Timing matters too. The site notes that rolling-shutter skew makes the left and right images describe different instants row by row, degrading the disparity match during fast motion, and that a global shutter on both cameras keeps the frame geometrically consistent. Depth quality is therefore set by calibration, shutter and sync behaviour, and a matching step the site does not describe.

The chain the site describes: capture a left and right pair, apply the calibration measured that session, recover per-pixel distance from the disparity, register it to the head-camera frame, and ship it beside RGB, IMU and hand pose on one PTP clock with int64 nanosecond timestamps.

Depth is metric millimetres: streams/depth_mm in HDF5 (uint16, 480 × 848 at 30 Hz) with a per-pixel confidence map in streams/depth_conf, observation.depth_mm in RLDS, and a 16-bit depth.png in WebDataset. Intrinsics, extrinsics (SE3) and rms_reproj_px sit in calibration/, and each rig session is checked against rms_reproj_px under 0.28.

Disparity is not a default stream. Raw frames ship with calibration for re-rectification, and rectified pairs plus disparity can be added on request. No disparity field name, data type or resolution is stated, so define them in the spec.

Start with calibration: confirm every episode ships intrinsics, extrinsics and the baseline, plus a reprojection-error figure. Firsthand’s stated check is rms_reproj_px under 0.28 per rig session.

Then check the depth encoding (metric units, integer type, low-confidence marking) and sync with the video. Firsthand depth is uint16 millimetres with a confidence channel, and the site warns against decoding the WebDataset PNG as 8-bit. Finally, baseline length, focal length, depth range, depth accuracy, disparity range and matching algorithm are not stated, so put them in the spec as acceptance criteria.

What is stereo disparity in simple terms?

It is how far a point appears to shift horizontally between the left and right images of a stereo pair, measured in pixels. Nearer points shift more and distant points shift less, so the shift tells you how far away the point is.

What is the formula for depth from disparity?

Depth = focal length × baseline / disparity. The OpenCV stereo depth tutorial writes it as disparity = B·f / Z and notes that depth is inversely proportional to the difference between corresponding image positions.

Is a disparity map the same as a depth map?

No. A disparity map stores pixel shifts; a depth map stores distances. You convert one into the other with the calibrated baseline and focal length, and pixels that failed to match have no reliable depth in either.

Does Firsthand deliver disparity maps?

Depth is the default stereo output, delivered in metric millimetres. The site states that rectified pairs plus disparity can be added to the delivery on request. It does not state a disparity field name, data type or resolution.

What baseline does the Firsthand stereo rig use?

A baseline close to human interpupillary distance. The site gives no baseline length in millimetres, so ask for the figure in your collection spec or read it from the per-episode calibration.

How accurate is Firsthand stereo depth?

Not stated. The site states no depth accuracy figure. What it does state is that calibration is measured every session, that each rig session is checked against rms_reproj_px under 0.28, and that a confidence channel flags low-confidence pixels.

Can I request a custom egocentric collection with stereo depth?

Yes. Firsthand runs custom collections against a written spec, with depth, calibration and sync shipped with each episode. See calibrated egocentric video data, then request a custom dataset.