Calibrated Egocentric Video Data

Calibrated egocentric video data is first-person task footage delivered together with the calibration of the cameras that recorded it. On Firsthand’s rig that means per-camera intrinsics (focal length, principal point and lens distortion coefficients), extrinsics that place the second head camera and both wrist cameras relative to the head camera, depth registered to the head-camera frame, and per-stream timestamps on one PTP clock.

This page sells the collection: fast, on-demand, custom collection scoped to your skill spec, with calibration delivered as part of every episode. It is not the how-to and it is not the definition. The guide on calibrating head and wrist cameras teaches the method, and the camera calibration glossary entry defines the terms. Here the focus is what you receive and how to order it.

A manipulation policy has to relate what the head cameras see to what the wrist cameras see. The site’s calibration guide puts it this way: what matters is knowing exactly how a pixel in one camera’s frame relates to a pixel in another camera’s frame, so that hand pose, object pose and depth all agree across every stream.

The site’s glossary states the cost of getting it wrong: uncalibrated or drifted camera parameters make depth, hand pose and object pose unreliable, because triangulation and rectification depend on knowing where each camera is and how its lens distorts the image. Hand joints, for example, are recorded in metres in the head-camera frame, and that only means what it claims if the intrinsics and extrinsics are accurate.

The site states the dependency but does not quantify how much it changes model performance, and this page does not either.

Each episode’s calibration record follows the calibration/ group of the episode schema: intrinsics, extrinsics (SE3) and rms_reproj_px. Intrinsics are solved independently for every camera: fx, fy, cx, cy and the distortion coefficients k1, k2, p1, p2, k3. Extrinsics are a 4×4 SE(3) pose per camera, with the head-camera frame as the identity and every other camera expressed relative to it.

Depth is registered to the head-camera frame using the same solved extrinsics as the video streams. The stereo baseline, the fixed distance between the two lenses of the head pair, is measured every session and shipped with the episode. The first table below lists each field, what the site states about it and what it is used for.

The capture page describes the procedure. A checkerboard target, 9×6 with 30 mm squares, is recorded in 40 poses spanning the full frustum of every camera. Intrinsics are solved per camera, and the session does not start unless RMS reprojection lands under 0.28 px. Extrinsics are then solved against the shared target and expressed in the head-camera frame. IMU alignment estimates gyro bias and accelerometer scale from a 60 s static hold plus a figure-eight excitation.

The entire procedure repeats at the end of the session. The difference between the opening and closing extrinsics is the drift for that episode, written into episode metadata as calib_drift_mm. Both solves ship with the episode, so you can re-run the reprojection check on your own copy of the data rather than relying on a pass or fail label.

Calibration says where each camera is; synchronization says when it fired. The guide states that both have to hold, because a calibrated extrinsic between two cameras only describes the scene correctly if both captured it at the same instant. Firsthand runs the head-mounted stereo pair, both wrist cameras, the depth sensor and the head and wrist IMUs on one PTP clock domain, with a PTP grandmaster in the belt pack and a hardware trigger line to every sensor.

Every delivered stream carries int64 nanosecond timestamps in that domain, so sync error can be recomputed from the data. The stated figures are a 1.12 ms median and 1.94 ms p99 against a 2.0 ms ceiling; the capture page lists any stream exceeding the ceiling for more than three consecutive frames as a reject reason.

The rig the site describes has a global-shutter head camera at 4K and 60 fps with a 120° horizontal field of view, two wrist cameras at 1920×1080 and 60 fps, a depth stream at 848×480 and 30 fps in metric millimetres, and 6-DoF IMUs on the head and both wrists at 200 Hz. The second table maps each stream to its role in the calibration. Streams beyond RGB are added where the spec calls for them.

The episode schema is the canonical layout and every delivery format is a projection of it. In HDF5 and zarr, the calibration/ group mirrors the schema and carries both the opening and closing solves for drift checks. In RLDS, episode_metadata.calib_drift_mm lets you filter on calibration quality before training. The head-camera frame is the identity for all extrinsics in the schema.

The guide notes consistent stream naming across HDF5, RLDS and WebDataset delivery. How the full calibration record is laid out in LeRobot and WebDataset deliveries is not stated beyond the format pages, so it is set per program in the skill spec. Name the format you load from in the spec, and the calibration layout is agreed with it.

Delivered calibration changes what your team does before training, not what your model is. With it, a team can undistort frames using the delivered coefficients, transform wrist-camera hand poses into the head-camera frame, check depth alignment, and filter episodes on calib_drift_mm. Without it, each of those steps depends on geometry the team has to assume or re-estimate.

The third table sets the two side by side. It states what each step needs, not a measured accuracy gain; the site publishes no such benchmark, and none is claimed here.

Calibration is part of the validated hour. The site defines a validated hour as 60 minutes of footage that has passed QA against your spec: sync within the ceiling, calibration solved and within drift tolerance, coverage cells filled and consent documented. Calibration drift, where the closing solve differs from the opening solve beyond tolerance, is a listed reject reason.

Rejected episodes go to the reject log and are recaptured at the provider’s cost. Billing is per validated hour, and the site states that rejected episodes, recaptures, calibration time and travel are never billed.

Some calibration details are not fixed on this site and are agreed in the skill spec instead. The last table separates what the site states from what is set per program.

Send the tasks, the environments and regions, the streams and labels, and the delivery format you need, and name the calibration outputs you want in the spec. If you have no written spec yet, the first draft is written with you on a call. The site states the order flow as spec, quote, Spec Pilot, scale, with the quote scoped against the spec.

Where coverage for your tasks and regions is deep, the median time from signed spec to first validated batch is 48 hours; where it is thin, turnaround runs three to four weeks. Delivered data ships under a perpetual, buyer-owned license.