Egocentric Hand Tracking Data Collection

Egocentric hand tracking data is first-person video with the 3D position of every hand and finger joint tracked frame by frame, delivered as a per-frame joint skeleton rather than a bounding box or a single wrist keypoint. The viewpoint is first-person because that is the viewpoint a robot’s own camera, or a head-mounted capture rig standing in for it, actually sees from during manipulation.

The distinction from hand pose estimation in general is narrow but matters for sourcing: hand pose estimation is the capability, egocentric hand tracking data is that capability’s output captured specifically from the operating viewpoint a manipulation policy will be deployed with, rather than from a fixed external camera watching the scene from the side.

Manipulation and dexterous-hand policies condition on where the fingers are relative to the object being handled, not just that a hand is present in frame. A 2D bounding box or a single wrist keypoint throws away exactly the signal a grasp, regrasp, or in-hand reorientation needs to learn from: which fingers are in contact, how the hand is curled around the object, and how that shape changes through the motion.

First-person capture keeps the hand and the object it is manipulating in frame from the viewpoint a policy will actually operate in, which third-person footage and most public hand-pose datasets, captured from a fixed external camera, do not provide. A model trained on third-person hand pose has to additionally learn the viewpoint transform at inference time; training on egocentric hand tracking data removes that gap.

This matters most for VLA and dexterous-hand pretraining, where the policy’s action space is defined in terms of finger and wrist motion rather than a coarse end-effector pose, and for any manipulation task where grasp quality rather than just reach-and-place success is being measured.

A head-mounted camera keeps the hands and the manipulated object in frame through the full reach-grasp-adjust cycle. Depth, IMU, and hand pose are captured on one hardware-triggered clock, so every stream lines up to the same timeline rather than being stitched together after the fact from separately timestamped sensors.

A wrist-mounted camera can be added alongside the head view for close-range detail when the grasp itself, rather than the full-arm reach, is what matters to the policy. Pairing head and wrist views is the same setup used across this site’s manipulation and keypoint collections, so a buyer ordering hand tracking data can add the same wrist view they would order for any other manipulation dataset without a separate rig.

Stereo capture adds binocular depth for finer 3D geometry at contact, which is useful when the policy needs to reason about fine finger-to-surface distance rather than coarse hand position. Whether to include wrist camera, stereo, or both is a spec decision made before capture starts, not a fixed default.

Hands occlude themselves and are occluded by the objects they manipulate constantly during real tasks: a finger curled behind the object it is gripping, or one hand passing behind the other during a bimanual task, is the normal case rather than the exception. Rather than letting a tracker interpolate or hallucinate a pose for a hidden joint, each joint carries a visibility flag (hand_visible) so a model can distinguish an observed position from an occluded one.

This is the same per-joint visibility flag used across every hand-pose and keypoint annotation on this site, so hand tracking data is consistent with the hand pose, contact, and keypoint annotation layers a buyer might also order, and the two can be combined in the same dataset without reconciling two different occlusion conventions.

Marking occlusion explicitly, rather than silently filling it in, is also what makes the data usable for training rather than just visualization: a policy learning from a position that was actually hallucinated through occlusion learns a wrong signal with no way to tell it apart from a real observation.

Hand tracking labels go through the same QA process every annotation layer on this site goes through: review against the synchronized depth and video it was tracked from, with clips that fail acceptance criteria rejected and recaptured rather than shipped with a flagged caveat. Acceptance criteria are agreed as part of the spec before capture scales, so both sides know what "passes" means before volume is committed.

Because hand joints are tracked against depth rather than estimated from RGB alone, reviewers can check 3D plausibility — does the joint position make geometric sense against the depth map, is the hand the right size and distance from the camera — not just whether a skeleton looks reasonable on a 2D frame. That is a stronger check than most RGB-only hand pose pipelines can run, because there is a second, independent measurement (depth) to validate the tracked joints against.

Public hand-pose and egocentric datasets are useful for establishing a general prior, but they were collected for someone else’s tasks, scenes, and camera setup. A custom hand tracking collection lets you specify the exact tasks, objects, environments, and contributor diversity a policy needs, with depth and visibility flags matched to your own capture rig rather than a published dataset’s.

Custom collection is the better fit when a program needs coverage of a specific task set, a particular hand or tool, or occlusion patterns that a general-purpose public dataset does not represent well, or when the license terms of a public dataset do not permit the downstream commercial use a policy is being trained for.

The two are not mutually exclusive: a common pattern is to pretrain on a broad public prior for general hand-motion statistics, then fine-tune on a smaller custom collection scoped tightly to the target task set, objects, and environment, which is how this site’s pretraining-data-collection pages frame the general pattern.

Hand tracking data ships in RLDS, LeRobot, WebDataset, or HDF5, matching the training stack a buyer already uses. In the LeRobot v2.1 layout, hand joints are flattened and packed into observation.state, hands first, then body joints if ordered, in a fixed documented ordering. In RLDS and WebDataset, hand_joints and hand_visible are carried as explicit per-step fields alongside RGB, depth, and action labels.

Because the same underlying hand_joints and hand_visible fields are reused across every format, a buyer who starts in one training stack and later moves to another is not locked into re-collecting data; the export layout changes, the underlying 3D joint data does not.

Ordering starts the same way any custom collection on this site is scoped: a written spec covering the tasks, objects, and environments the hand tracking data needs to cover, which hand or hands are involved, contributor diversity targets, and whether wrist camera, stereo depth, body pose, or contact events should be added alongside the base head-mounted capture.

From there the engagement follows the same pilot-then-scale pattern used across this site’s custom collection: a small first batch proves the spec and acceptance criteria before volume is committed, and scaling happens on demand against the agreed spec rather than as a single fixed-size delivery.