What Is Hand-Eye Calibration?

An egocentric rig has more than one eye. The head view shows where the person is looking, and the wrist views stay close to the hand. A model has to know how a point in one view relates to the same point in another, or hand pose, object pose and depth disagree across streams. The site states that this relationship is extrinsic calibration: a 4×4 SE(3) transform per camera in one shared frame. Calibrating each camera alone gives its lens distortion, not where it sits relative to the others.

Firsthand states a head-mounted global-shutter camera plus two wrist cameras and a depth sensor. The head-to-wrist-to-depth transforms are solved against a shared calibration target and expressed in the head-camera frame, which is the identity, so every other camera is a 4×4 SE(3) pose relative to it. Calibration runs twice per episode, before and after capture, because a worn head mount can flex and a wrist camera can shift. The site does not describe this as an AX = XB solve or name a solver. It is multi-camera extrinsic calibration, closely related to hand-eye calibration but not OpenCV’s gripper-to-camera formulation.

Hand-eye calibration finds the fixed rigid transform between a camera, the eye, and the hand or gripper it is mounted on. OpenCV’s documentation describes calibrateHandEye as computing the camera-to-gripper transformation for a camera mounted on a robot gripper, which it calls eye-in-hand. It calls a static camera watching a pattern on the end-effector eye-to-hand.

Because the transform is fixed, wherever the hand moves, the camera pose follows it. The problem is also known as solving AX = XB, where X is the unknown camera-to-gripper transform.

OpenCV describes the procedure. A static calibration pattern gives the target-to-camera transform at each pose, for example by PnP from 2D-3D correspondences. The gripper is moved to acquire several poses, and the gripper-to-base transform is recorded each time, for example from robot kinematics. The function then returns the camera-to-gripper rotation and translation.

OpenCV implements five methods: three that estimate rotation and then translation, and two that estimate both at once. It states that at least 2 motions with non-parallel rotation axes are needed, so at least 3 different poses, and strongly recommends many more.

The check the site states is RMS reprojection error: project the known target points back into each camera and measure how far they land from where they were observed. Firsthand gates every episode on RMS reprojection under 0.28 px, checked at both the opening and closing solve.

A second check catches a rig that calibrated well and then shifted during the session: drift between the opening and closing extrinsics. An episode ships only if drift is within tolerance, and failures go to the reject log. The tolerance value is not stated.

Each episode carries a calibration/ group with intrinsics, extrinsics (SE3) and rms_reproj_px. Both the opening and closing solves ship, so a buyer can re-run the reprojection check on their own copy. Measured drift is carried as calib_drift_mm in the episode metadata, so you can filter on calibration quality before training.

The site does not state the transform direction or translation units, so confirm your loader treats the head-camera frame as the identity and test the path on a pilot batch.

Custom and on-demand collections are scoped by a written skill spec. Put the calibration requirements in it: which calibration fields you need delivered, which checks you will re-run, and any tolerance tighter than the stated 0.28 px gate. The site lists fields, not numeric calibration values, so anything beyond them is agreed in the spec. Request a custom dataset to start a pilot.

What is hand-eye calibration?

It solves the fixed transform between a camera and the gripper or hand it is mounted on. OpenCV’s calibrateHandEye returns the camera-to-gripper transform by solving AX = XB.

What are head and wrist camera extrinsics?

They are the 4×4 SE(3) poses that place each camera relative to the head camera. Firsthand states the head-camera frame as the identity, with the wrist cameras expressed relative to it.

Does Firsthand use OpenCV’s calibrateHandEye?

Not stated. The site describes extrinsics solved against a shared calibration target and gated on reprojection error, but names no solver or AX = XB formulation.

How does Firsthand check its calibration?

Each episode is gated on RMS reprojection under 0.28 px at the opening and closing solves, and open-to-close drift must be within tolerance. Failures go to the reject log.

Can I re-check the calibration myself?

Yes. Both solves ship in the calibration/ group with rms_reproj_px, and drift is carried as calib_drift_mm, so you can re-run the reprojection check on your own copy.

Can I request custom collection with specific calibration requirements?

Yes. Custom and on-demand collections are scoped by a written skill spec, so list the calibration fields and checks you need and request a custom dataset to start a pilot.