Keypoint and pose annotation for robot learning.

A keypoint is a named point on a body or object — a wrist, a knuckle, a fingertip. Pose annotation labels where those keypoints are in every frame of a video, producing a skeleton that moves with the person. Human activity recognition, motion retargeting and robot learning from human video all rely on it.

Keypoints can be labeled in 2D, as pixel coordinates on the image, or in 3D, as positions in space. 2D labels are what most general-purpose annotation tools produce. They are enough to draw a skeleton over a frame, but they do not say how far away a joint is, and a robot hand has to reach a real point in space, not a pixel.

Manipulation happens in metric space. A grasp succeeds or fails on centimetres, and a policy that learns from human hands has to map those hands onto its own end-effector. A 2D keypoint is ambiguous along the camera’s line of sight: the same pixel could be a fingertip close to the lens or one much further away.

Firsthand annotates hand pose against the synchronized depth stream captured on the same hardware clock as the video. Joint positions reflect real geometry rather than being estimated from a single 2D frame, and they are stored in metres in the head-camera frame, which is the identity for every extrinsic in the calibration group. That makes the pose directly comparable with depth, wrist-camera views and IMU data without any per-episode re-registration.

In first-person manipulation footage the hands occlude themselves constantly. Fingers curl behind a mug, one hand crosses the other, and the moment of contact is often the moment a joint disappears from view. An annotation pipeline that forces a coordinate for every joint in every frame will fill those gaps with guesses.

Firsthand carries a per-joint visibility flag, hand_visible, alongside every joint position. A training pipeline can then weight, mask or drop occluded joints explicitly instead of learning from invented positions. The wrist cameras help here too: they keep the grasp point in frame when the head camera’s view is blocked.

Every stream carries its own int64 timestamp in the PTP domain, and streams are stored at their native rate: head and wrist video at 60 Hz, depth at 30 Hz and IMU at 200 Hz. Loaders resample at read time, never in the delivery, so no information is thrown away to force a common rate.

Calibration ships with every episode as intrinsics, SE3 extrinsics and the RMS reprojection error, which is held below 0.28 pixels. That is the number that tells you whether a 3D joint projected back into a wrist-camera frame will land where the hand actually is.

Pose annotation on Firsthand is part of a custom collection rather than a stand-alone labeling job on third-party footage, because 3D joints depend on the synchronized depth and calibration captured at the same time. A skill spec defines the task, environment, the geometry that must stay in frame and the acceptance test; contributors capture against it; annotation and QA run per batch; and the episode ships in your format.

First validated batches arrive in a median of 48 hours where coverage is deep and three to four weeks where contributors need to be recruited first. Billing is per validated hour, and takes that fail QA are rejected and recaptured at Firsthand’s cost.