What Is Stereo Matching?

Every stereo depth value is the result of a match. A wrong match gives a wrong depth, and a pixel with no accepted match gives none. Calibration and rectification make matching possible, but the matcher decides which pixels get a value at all, so stereo depth estimation depends on image texture, window settings and timing as well as on the rig.

Firsthand derives depth from a head-mounted, calibrated stereo rig, not from a standalone depth sensor, and delivers it as metric millimetres beside RGB, IMU and hand pose. The site does not state which matching algorithm produces that depth, so treat the method, depth range and accuracy as items to fix in a written spec for a custom, on-demand collection.

Two cameras mounted side by side see one scene from slightly different positions, so the same point lands at different places in each image. Stereo matching is the search that pairs those places up. The problem is called stereo correspondence, and OpenCV’s documentation uses that phrase for what its block-matching class computes. Stereo depth estimation is the wider task of turning the matches into distances.

The output of matching is a disparity map: one disparity value per pixel, wherever a match was accepted. It is an estimate, not a measurement of the scene. Pixels where no confident match was found are filtered out, and anything built on the map, including depth, inherits those gaps.

The OpenCV depth-map tutorial gives the sequence: the algorithm finds corresponding matches between the two images, and once it has matches it finds the disparity. It then writes the triangle geometry as disparity = x − x′ = B·f / Z, where B is the distance between the two cameras, f is the focal length and Z is the depth. In its words, the depth of a point is inversely proportional to the difference between its image positions.

Matching therefore supplies the one quantity that changes per pixel, the disparity, while calibration supplies the other two. Because depth varies with the inverse of disparity, a small matching error costs more for distant points than for near ones. The stereo disparity entry covers that relation in more detail.

OpenCV ships two matchers. Its StereoBM class is documented as computing stereo correspondence using the block matching algorithm, introduced and contributed to OpenCV by K. Konolige. It compares blocks of pixels along the row of the other image. The documentation says a larger block gives a smoother but less accurate disparity map, while a smaller block gives more detail but a higher chance of finding a wrong correspondence.

StereoSGBM implements a modified version of H. Hirschmuller’s algorithm, usually called semi-global matching. By default it is single-pass and considers 5 directions instead of 8, and it matches blocks rather than individual pixels. Two parameters, P1 and P2, penalise disparity changes between neighbouring pixels, and P2 must be larger than P1, which favours smooth surfaces. The documentation also warns that its full two-pass mode needs far more memory.

Both classes can post-filter their results, and both expose a number-of-disparities setting that defines the search range. The documentation describes the algorithms; it does not say which one any particular dataset uses.

In a raw pair, the match for a point can lie anywhere along a tilted epipolar line. OpenCV states that stereoRectify computes rotation matrices that make both image planes the same plane, which makes the epipolar lines parallel and simplifies the dense stereo correspondence problem. The depth tutorial adds that the epipolar constraint makes the search faster and more accurate.

After rectification, matching is a search along one row. The StereoSGBM reference notes that rectification can shift the images, so its minimum-disparity parameter may need adjusting. Firsthand ships raw frames with calibration so a buyer can re-rectify, and states that rectified pairs plus disparity can be added on request. The stereo rectification entry explains the transform itself.

Low texture is the classic failure. The OpenCV tutorial describes a texture threshold that filters out areas without enough texture for reliable matching, and the StereoBM reference describes it as removing regions where establishing correspondences is difficult. A blank surface offers the matcher nothing to compare.

Ambiguity is the second. A uniqueness ratio removes a pixel when its best match is not sufficiently better than the alternatives. Boundaries are the third: the tutorial notes that block matchers produce speckles near object edges, where the matching window catches the foreground on one side and the background on the other, and that a speckle filter removes them.

Capture conditions matter as well. The site states that rolling-shutter skew makes the left and right images describe different instants row by row, which degrades the match during fast motion, and that a global shutter on both cameras keeps the frame geometrically consistent. Calibration error shifts every converted depth value, so a clean match can still give a wrong distance.

The chain the site describes is: capture a left and right pair from the head-mounted rig, apply the calibration measured that session, recover per-pixel distance from the disparity, register it to the head-camera frame, and ship it beside RGB, IMU and hand pose on one PTP clock. Each rig session is checked against rms_reproj_px under 0.28.

Depth is metric millimetres: streams/depth_mm in HDF5 (uint16, 480 × 848 at 30 Hz) with a per-pixel confidence map in streams/depth_conf, observation.depth_mm in RLDS, and a 16-bit depth.png in WebDataset. The confidence channel is where unmatched or low-confidence pixels show up for a buyer.

Firsthand runs custom collection as a written-spec, on-demand service. Put the stereo requirements in that spec: whether you want rectified pairs and disparity added, how low-confidence pixels should be handled, and the depth range and accuracy you need, since the site states neither.

What is stereo matching in simple terms?

It is the search that finds the same scene point in the left and right images of a stereo pair. The horizontal shift between the two positions is the disparity, and once calibration turns that shift into distance you have depth.

Is stereo matching the same as stereo correspondence?

Yes, in practice the two terms name the same problem. OpenCV’s StereoBM reference describes its algorithm as computing stereo correspondence using block matching, and the depth tutorial describes the same step as finding corresponding matches between two images.

What is the difference between block matching and semi-global matching?

Block matching compares blocks along the other image’s row. Semi-global matching, as OpenCV implements it, also penalises disparity changes between neighbouring pixels through P1 and P2, which favours smooth surfaces. OpenCV documents both as separate classes with their own filtering steps.

Why does stereo matching struggle on blank surfaces?

There is no texture to compare. OpenCV describes a texture threshold that filters out areas without enough texture for reliable matching, so featureless regions end up with no reliable disparity and therefore no reliable depth.

Which matching algorithm does Firsthand use?

Not stated. The site states that depth is derived from the calibrated stereo rig, but it does not name the matching method, so ask for it in your collection spec if your pipeline depends on it.

Does Firsthand deliver stereo depth?

Yes. Depth is delivered in metric millimetres as streams/depth_mm with a streams/depth_conf confidence map in HDF5, observation.depth_mm in RLDS, and a 16-bit depth.png in WebDataset. Rectified pairs plus disparity can be added on request.

Can I request a custom egocentric collection with stereo depth?

Yes. Firsthand runs custom collections as a written-spec, on-demand service, with depth, calibration and sync shipped with each episode. See stereoscopic video data collection, then request a custom dataset.