What Is Stereo Rectification?

A stereo pair only yields depth if the same scene point can be found in both images. Rectification makes that search consistent and is the step that gives disparity its meaning. For a robotics data buyer the practical risk is a mismatch: if a pair is rectified with the wrong calibration, or treated as rectified when it is still raw, the disparity computed from it is wrong, and the error carries straight into depth.

Firsthand ships raw stereo frames with the calibration measured that session, so a buyer can rectify the pair themselves, and rectified pairs plus disparity can be added to the delivery on request. The calibration sits in calibration/ as intrinsics, extrinsics (SE3) and rms_reproj_px, with the head-camera frame as the identity, and each rig session is checked against rms_reproj_px under 0.28. The site states no rectification algorithm, rectified resolution, rectification error or rectification-map field.

Two cameras mounted side by side see a scene from slightly different positions. In the raw images a scene point can land at a different height as well as a different horizontal position, because the cameras are never perfectly aligned and each lens bends the image. Stereo rectification is the transformation that removes that misalignment. It warps both images, using the calibrated parameters, so they behave as if they came from two parallel cameras, and a point in the left image appears on the same row in the right image.

OpenCV, a common reference implementation, describes its stereoRectify function as computing rectification transforms for each head of a calibrated stereo camera. Its documentation introduces the step as the transformation that makes the camera optical axes parallel.

In a raw pair, the match for a point in the left image lies somewhere along an epipolar line in the right image, and that line can be tilted and differ from row to row. OpenCV’s documentation states that stereoRectify computes rotation matrices that (virtually) make both camera image planes the same plane. That makes all the epipolar lines parallel and so simplifies the dense stereo correspondence problem.

For the common horizontal layout, the documentation states that the corresponding epipolar lines in the left and right cameras become horizontal and have the same y-coordinate. That is the payoff: after rectification, matching is a one-dimensional search along a row, and the difference in horizontal position is the disparity.

Rectification needs three groups of values. Intrinsics describe each camera’s own lens and sensor: focal lengths, the principal point and the distortion coefficients. Extrinsics give the rotation and translation between the two cameras. The distance between the lenses, the baseline, belongs to the same calibration and later scales disparity into metric depth. In OpenCV, stereoCalibrate finds the intrinsic parameters for each camera and the extrinsic parameters between them, and stereoRectify takes the matrices it computes as input.

The site lists the same ingredients for Firsthand: intrinsics (fx, fy, cx, cy and lens distortion coefficients), extrinsics as a 4×4 SE(3) pose per camera with the head-camera frame as the identity, and the baseline, all measured each session.

Disparity is the horizontal pixel shift of the same point between the rectified left and right images, and depth then follows from focal length times baseline divided by disparity. The OpenCV stereo depth tutorial writes the same relation as disparity = B·f / Z. Because that relation assumes rectified geometry, a pair that is still raw cannot be fed straight into it.

The site states that Firsthand derives depth from its calibrated stereo rig and delivers it in metric millimetres, registered to the head-camera frame. It does not state the matching algorithm, depth range or depth accuracy.

The stated position is specific. Raw frames ship with calibration so you can rectify the pair yourself, and rectified pairs plus disparity can be added to the delivery on request. The default is therefore raw frames plus calibration, and rectified pairs are something you request in the spec.

The calibration is the same block every episode carries: calibration/ holds intrinsics, extrinsics (SE3) and rms_reproj_px. An episode ships only if RMS reprojection stays under 0.28 px and the drift between the opening and closing calibration solves is within tolerance. That figure is a reprojection-error gate, not a rectification-error figure, and the site states no rectification error. Depth arrives separately as streams/depth_mm (uint16, 480 × 848 at 30 Hz) with a streams/depth_conf confidence map. The site states no field names for rectified frames or rectification maps.

Confirm that every episode ships intrinsics, extrinsics and the baseline, plus a reprojection-error figure. Firsthand’s stated gate is rms_reproj_px under 0.28. Check that the reference frame is documented; here the head-camera frame is the identity for all extrinsics.

Then decide in the spec whether you will rectify from raw frames yourself or want rectified pairs plus disparity added, and write down the field names, data type and resolution you expect, since the site does not state them. Make baseline length, focal length, rectified resolution, rectification error and depth accuracy acceptance criteria, because none is stated.

What is stereo rectification in simple terms?

It is a warp applied to the left and right images of a calibrated stereo pair so that a point seen in both lands on the same row. Once rows line up, finding the match is a search along one row instead of across the image.

Why is rectification needed before computing disparity?

Disparity is defined as the horizontal shift of a point between the rectified left and right images. Rectification makes that shift purely horizontal. OpenCV states that it makes the epipolar lines parallel and so simplifies the dense stereo correspondence problem.

Does Firsthand deliver rectified stereo frames?

Raw frames ship with calibration so you can rectify the pair yourself, and rectified pairs plus disparity can be added to the delivery on request. Rectified pairs are not stated as a default, so ask for them in the spec.

What calibration data does rectification need?

Each camera’s intrinsics (focal lengths, principal point, distortion coefficients) and the extrinsics between the cameras. Firsthand states that intrinsics, extrinsics (SE3) and rms_reproj_px ship in calibration/, with the baseline measured every session.

How accurate is the rectification?

Not stated. The site states no rectification error. Its stated check is rms_reproj_px under 0.28 per rig session, which is a reprojection-error gate and not a rectification-error figure.

Can I request a custom egocentric collection with rectified stereo?

Yes. Firsthand runs custom collections against a written spec, so state whether you want raw frames plus calibration or rectified pairs plus disparity, and define the field names, type and resolution you expect. See calibrated egocentric video data, then request a custom dataset.