What Are Camera Intrinsics and Extrinsics?

Depth, hand pose and object pose all depend on knowing how each lens maps rays to pixels and where each camera sits. Assumed values, or values that drift during a session, corrupt that geometry.

The site states per-episode calibration/ intrinsics (fx, fy, cx, cy, k1, k2, p1, p2, k3) and extrinsics (4×4 SE(3), head-camera frame as identity) with rms_reproj_px under 0.28. Numeric values are not stated.

A camera turns 3D points into 2D pixels, and two sets of numbers describe that mapping. OpenCV’s calibration documentation builds both into one pinhole model, s p = A [R|t] P_w, where A is the intrinsic matrix and R and t are the extrinsic rotation and translation.

Calibration solves these values from a known pattern. OpenCV’s calibrateCamera finds intrinsic and extrinsic parameters from several views of a calibration pattern, and stereoCalibrate finds each camera’s intrinsics plus the extrinsics between the two cameras.

OpenCV describes the intrinsic matrix as the focal lengths fx and fy, in pixel units, and the principal point (cx, cy), usually close to the image centre. Real lenses also distort the image, mostly radially with slight tangential distortion, and OpenCV counts the distortion coefficients (k1, k2, p1, p2, k3 and further terms) as intrinsic parameters too.

Intrinsics do not depend on the scene, so once estimated they can be reused while the focal length is fixed. If an image is scaled, OpenCV says fx, fy, cx and cy must be scaled by the same factor, while the distortion coefficients stay the same regardless of image resolution.

Extrinsics are a 3×3 rotation R and a 3×1 translation t. In OpenCV’s model they change coordinates from the world frame to the camera frame, and they pack into one 4×4 homogeneous transform. Because they describe a pose, they change if the camera moves relative to its reference frame.

OpenCV’s stereoRectify takes each camera’s intrinsic matrix and distortion coefficients plus the rotation and translation between the cameras, and computes rectification transforms for each head of a calibrated stereo camera. Intrinsics supply each lens’s distortion and pixel-to-ray geometry; extrinsics place the two views relative to each other.

The site’s camera calibration entry states the cost of getting them wrong: drifted parameters make depth, hand pose and object pose unreliable, because triangulation and rectification depend on knowing where each camera is and how its lens distorts the image.

Each episode carries a calibration/ group with intrinsics, extrinsics (SE3) and rms_reproj_px. Intrinsics are listed as fx, fy, cx, cy, k1, k2, p1, p2, k3 per camera. Extrinsics are a 4×4 SE(3) pose per camera, with the head-camera frame as the identity. The rig is a head-mounted global-shutter camera plus two wrist cameras and a depth sensor, and depth is registered to the head-camera frame using the same solved extrinsics as the video streams.

Calibration is solved twice per episode, before and after capture, and both solves ship in calibration/. An episode ships only if RMS reprojection stays under 0.28 px and open-to-close drift is within tolerance, with drift carried as calib_drift_mm in the episode metadata. Raw frames ship with calibration so you can re-rectify, and rectified pairs plus disparity can be added to the delivery.

Confirm every episode ships both solves and rms_reproj_px, and that your loader treats the head-camera frame as the identity. Check the transform direction, translation units and distortion-coefficient order against your own toolchain, because the site does not state them.

Filter on calib_drift_mm before training and re-run the reprojection check on the opening and closing solves yourself. Write any tighter threshold, required baseline or rectification check into the spec, then test it on a pilot batch.

What are camera intrinsics?

They are one camera’s own lens and sensor parameters: focal lengths fx and fy, the principal point (cx, cy) and distortion coefficients.

What are camera extrinsics?

They are the rotation and translation that place a camera relative to another frame. OpenCV packs them into a 4×4 transform that changes coordinates from the world frame to the camera frame.

What is the difference between intrinsics and extrinsics?

Intrinsics describe the camera itself and stay fixed while the focal length is fixed. Extrinsics describe the camera’s pose, so they change when it moves relative to its reference frame.

Why does stereo depth need both?

OpenCV’s stereoRectify takes each camera’s intrinsics and distortion plus the rotation and translation between the cameras. Intrinsics model each lens; extrinsics relate the two views.

Which calibration files does Firsthand deliver?

Each episode has a calibration/ group with intrinsics, extrinsics (SE3) and rms_reproj_px, from an opening and a closing solve. Episodes ship only under 0.28 px RMS reprojection with drift within tolerance.

Does Firsthand state focal length, baseline or distortion values?

Not stated. The site lists the fields, not the numbers, and describes the baseline only as human interpupillary. To require specific values or deliverables, write them into a spec and request a custom dataset.