How to Calibrate Head and Wrist Cameras Together

A head-mounted stereo pair and two wrist-mounted cameras each solve a different part of the same scene. The head cameras establish where the robot or operator is looking; the wrist cameras capture close-range manipulation detail at the hand. Neither view is useful on its own for training a manipulation policy — what matters is knowing exactly how a pixel in one camera’s frame relates to a pixel in another camera’s frame, so that hand pose, object pose, and depth all agree with each other across every stream.

That relationship is extrinsic calibration: a 4×4 SE(3) transform per camera, all expressed in a shared reference frame. Firsthand solves this with the head camera frame as identity and every other camera—the second head camera in the stereo pair, and both wrist cameras—expressed relative to it. Calibrating each camera in isolation would tell you its own lens distortion, but it would not tell you where that camera sits relative to the others, which is the number that actually lets hand_joints recorded in the head-camera frame mean what they claim to mean.

See /guides/camera-calibration-procedure for the full per-session procedure this page builds on, and /glossary/camera-calibration for the underlying definitions of intrinsics and extrinsics.

Intrinsic calibration solves a single camera’s own lens and sensor properties: focal length, optical center, and lens distortion coefficients. It answers "how does this camera, on its own, map 3D points onto its 2D sensor?" Every camera on the rig — both head cameras and both wrist cameras — gets its own independent intrinsic solve: fx, fy, cx, cy, and the distortion coefficients k1, k2, p1, p2, k3.

Extrinsic calibration solves where each camera sits in 3D space relative to the others. It answers "if a point appears at this pixel in the head camera, where does the same physical point appear in the wrist camera?" That is the transform that lets a dataset stitch together what the head sees and what the wrist sees into one consistent 3D scene, rather than four cameras each reporting their own disconnected 2D view.

Calibration solves where each camera is; synchronization solves when each camera fired. Both have to hold for head and wrist cameras to agree with each other, because a calibrated extrinsic transform between two cameras only describes the scene correctly if both cameras captured that scene at the same instant. A perfectly calibrated rig with cameras firing at slightly different times will still produce hand poses that do not line up with what the head camera sees.

Firsthand runs every sensor on the rig — the head-mounted stereo pair, both wrist cameras, the depth sensor, and the head- and wrist-mounted IMUs — on a single PTP (Precision Time Protocol) clock domain, with a PTP grandmaster in the belt pack and a hardware trigger line to every sensor. That is a different approach from relying on each camera’s own internal clock and timestamping frames after the fact in software, which drifts and cannot be independently checked.

Every delivered episode carries per-stream int64 nanosecond timestamps in that shared PTP domain, so the sync error between the head stream and either wrist stream can be recomputed directly from the data rather than taken on trust. See /capture for the reproducible sync-measurement method.

The IMUs sample at 200 Hz, faster than the camera frame rate, from three 6-DoF sensors — one head-mounted, one on each wrist. Because the IMU samples more often than the cameras capture frames, consuming the two streams together means resampling or windowing the IMU data to the frame you need, not assuming a 1:1 sample-to-frame mapping.

Firsthand brackets every capture session with two full calibration solves — one before capture, one after — rather than calibrating once and assuming the rig stays fixed for the rest of the session. A worn head mount flexes, and a wrist camera can shift slightly during a long manipulation session; bracketing both ends of the session turns that drift into a measured number instead of an unverified assumption.

• Opening solve — before capture starts, record a calibration target across the working volume visible to every camera, and solve intrinsics for each camera plus extrinsics in the head-camera frame, which is treated as identity.
• Capture the episode — run the task with every sensor hardware-triggered on the shared PTP clock; nothing about the rig geometry is changed mid-episode.
• Closing solve — immediately after capture, record the calibration target again and re-solve the same intrinsics and extrinsics.
• Measure drift — the difference between the opening and closing extrinsics is the drift for that episode, carried in the episode metadata as calib_drift_mm.
• Gate on the metrics — the episode ships only if RMS reprojection stays under 0.28 px and the measured drift is within tolerance; otherwise it goes to the reject log rather than shipping with unverified geometry.

Both the opening and closing calibration solves ship with the episode, so a buyer can re-run the reprojection check independently on their own copy of the data rather than relying on a pass/fail label alone.

The single number that matters most for verifying a multi-camera calibration is RMS reprojection error: take the solved intrinsics and extrinsics, project the known calibration-target points back into each camera’s image, and measure how far the projected points land from where they were actually observed. A low reprojection error across every camera — head and wrist alike — means the solved geometry actually explains what each camera saw; a high one means something in the solve is wrong, whether that is a bad target detection, a camera that moved during the target sweep, or a lens that was not properly modeled.

Firsthand’s acceptance gate is RMS reprojection under 0.28 px, checked at both the opening and closing solve. That threshold is applied automatically at the time of calibration — no human decides a borderline session is "probably fine." The second check, drift between the opening and closing solve, catches a different failure mode: a rig that calibrated well at the start but physically shifted during the session, which a single calibration pass at the start would never catch.

A dataset that only ships RGB and pose streams, with calibration quoted as a one-time spec on a data sheet, gives a buyer no way to check whether that calibration actually held for the specific episode they are training on. Firsthand ships both the opening and closing calibration solves with every episode: per-camera intrinsics, the per-camera extrinsic transforms in the head-camera frame, and the measured drift between the two solves as calib_drift_mm.

Everything the rig captures — head stereo RGB, wrist RGB, depth, and IMU — is delivered as separate, timestamped streams rather than pre-merged into a single track, with consistent stream naming across HDF5, RLDS, and WebDataset delivery.

See /formats/episode-schema and /formats for the full field-by-field breakdown across HDF5, RLDS, LeRobot-style step keys, and WebDataset shards.

hand_joints are recorded in metres in the head-camera frame. That only means what it claims to mean if the intrinsics for every camera and the extrinsics between them are measured accurately — a hand pose computed from a wrist camera has to be transformed into the head-camera frame through the solved extrinsic transform before it is comparable to anything the head camera sees. If that transform is wrong or has drifted, hand poses and object poses will be systematically offset in a way that is easy to miss by eye but corrupts a trained policy.

Depth inherits the same dependency: it is registered to the head-camera frame using the same solved extrinsics as the video streams, so accurate extrinsic calibration is what keeps depth geometrically aligned with the RGB and pose data captured alongside it. This is also why Firsthand checks calibration per rig session against the reprojection-error threshold, rather than assuming a one-time factory calibration holds indefinitely — the same drift that would misalign two camera views also misaligns the depth and hand-pose labels derived from them.

See /guides/how-we-measure-sync-error for how cross-stream timing error is measured and reported alongside calibration.

Teams that need egocentric data from a head-mounted stereo pair plus wrist cameras, calibrated and synchronized to this standard, can scope a custom collection with Firsthand rather than building and maintaining the rig, calibration pipeline, and acceptance gating in-house. The process starts with a skill spec, runs a small Spec Pilot sample through the same calibration and acceptance checks described here, and scales from there once the pilot is reviewed.

Because every episode already ships with both calibration solves and the per-stream PTP timestamps, a buyer evaluating a custom collection does not have to take calibration quality on faith — the reprojection and drift numbers are in the data itself. See /custom-collection/multiview-video-data-collection for how Firsthand scopes a synchronized head-plus-wrist multi-camera collection end to end.