Hand Pose and Wrist Action Labels for Robot Pretraining

An action label is the part of a training example that tells a policy what was done, as distinct from the video and depth that show what was seen. On a real robot, that label is usually a joint angle or gripper command, logged directly from the hardware. Human video has no robot in the loop, so there is no joint-angle log to record — the action has to be represented some other way if the footage is going to teach a policy anything about motion, not just appearance.

For egocentric human video, the two signals that stand in for a robot action log are wrist-level pose — the 6-degree-of-freedom position and orientation of the wrist over time — and hand pose, the per-joint 3D position of the fingers. Wrist pose gives the coarse trajectory: where the hand moved and how it was oriented on approach. Hand pose gives the fine structure: how the fingers were shaped around an object at the moment of contact. A pretraining corpus that has both can support both coarse reaching behavior and finer-grained manipulation behavior; a corpus with only one of the two can only support the behavior its label actually covers.

This page focuses on how action is represented in pretraining-scale human video and which of those fields Firsthand delivers on a custom collection. For the delivered episode layout, see /formats/episode-schema; for the fine-manipulation case for finger-level pose specifically, see /guides/3d-hand-pose-estimation-for-robotics.

NVIDIA’s EgoScale, documented on arXiv and on NVIDIA’s GEAR lab research page, pretrains on egocentric human video and represents the action in that video as wrist-level 6-DoF motion paired with articulated hand pose — keypoints across the joints of each hand — rather than a robot joint-angle log, since none exists in human footage. That pairing is what lets a corpus with no robot present still carry a usable action signal: wrist motion supplies the trajectory, hand keypoints supply the grasp shape.

As covered in /guides/what-is-nvidia-egoscale, EgoScale’s 20,854-hour corpus produced those labels with off-the-shelf SLAM for camera and wrist trajectory plus hand-pose estimation applied after capture, not measured live on a purpose-built rig. That is a deliberate scale-versus-precision tradeoff, and it is the reason this page draws a line between pseudo-labeled and captured action representations further down — both are wrist-plus-hand-pose in structure, but they differ in how the numbers were produced.

A wrist-level action label is embodiment-agnostic in a way a robot-specific joint-angle log is not. A 6-DoF wrist pose describes where a hand went and how it was oriented, independent of whether the thing doing the reaching is a human arm, a parallel-jaw gripper, or a 22-degree-of-freedom dexterous hand. That is what makes it usable as a pretraining signal at all: the same wrist trajectory that a human performed can be treated as a coarse target for very different downstream end effectors, because it does not encode any one gripper’s joint layout.

Hand pose is more embodiment-specific but still transfers further than a raw robot action would. Per-joint finger keypoints describe how a human hand shaped itself around an object; a downstream dexterous hand with a comparable number of degrees of freedom can be aligned to that shape more directly than a parallel gripper can, which is consistent with EgoScale’s own choice of a 22-DoF dexterous hand as its evaluation embodiment rather than a simpler gripper. This is also why /custom-collection/dexterous-manipulation-data and /custom-collection/dexterous-hand-pretraining-data both specify captured hand pose as a requirement, not an optional extra: the closer the target embodiment is to a human hand in degrees of freedom, the more directly the pretraining label needs to describe finger-level shape rather than just wrist trajectory.

The tradeoff runs the other way for coarse pick-and-place behavior on a simple gripper, where wrist-level pose alone can carry most of the useful signal and finer hand-pose detail contributes comparatively less. Which fields matter most is a function of the target embodiment and task, which is why a pretraining spec should state both the wrist and hand-pose requirement explicitly rather than assuming one implies the other.

Both approaches produce the same kind of label — a wrist pose and a set of hand keypoints per frame — but they differ in how those numbers are produced and in the error they carry as a result. A pseudo-label is computed after the fact from RGB using SLAM (for camera and wrist trajectory) and a hand-pose estimation model (for finger keypoints). A captured label is measured live during collection, from calibrated cameras and depth on a synchronized rig, at the moment the action happens.

Neither approach is wrong for its purpose. EgoScale’s own evaluation shows the pseudo-labeled pretraining recipe transferring to real task success, so scale-first pseudo-labeling clearly carries useful signal. The distinction matters for planning a mixture: a broad pretraining stage can tolerate pseudo-label noise better than a small, closely matched alignment stage can, which is the same two-stage structure EgoScale itself uses — large-scale pseudo-labeled pretraining, then a much smaller aligned stage. See /compare/egoscale for the full row-by-row comparison and /guides/building-a-robot-pretraining-data-mixture for how to plan the split.

A pretraining-ready episode needs more than a single wrist number or a single hand-pose number — it needs a small set of fields that, together, cover trajectory, grasp shape, contact, and occlusion. Firsthand’s own episode schema, documented in full at /formats/episode-schema, ships the following fields for exactly this purpose.

Wrist-mounted RGB cameras (streams/wrist_l, streams/wrist_r) are a separate optional stream from the wrist IMUs above — they add close-range visual detail for fine manipulation, distinct from the 6-DoF motion the IMU records. Both are documented in /formats/episode-schema.

A hand-pose or wrist label is only as useful as the frame it is attached to. If a hand-joint sample and the video frame it is meant to describe were captured a few milliseconds apart, the label describes a slightly different moment than the image shows — the hand may have already moved past the pose the label records. Firsthand’s rig hardware-triggers every stream, including the wrist IMUs and hand-pose tracking, onto the same PTP clock domain, so a video frame and its corresponding hand-pose or IMU sample carry matching timestamps rather than being aligned after the fact in software.

Calibration affects the same problem from a different angle: hand_joints are recorded in metres in the head-camera frame, which only means what it claims to mean if the camera’s intrinsics and the SE(3) extrinsics between streams are measured accurately. Firsthand checks that per rig session against a reprojection-error threshold (rms_reproj_px < 0.28) rather than assuming a one-time factory calibration holds indefinitely. See /guides/camera-calibration-procedure for the calibration process and /guides/how-we-measure-sync-error for how cross-stream sync error is measured and reported.

The underlying fields are the same across delivery formats — hand_joints, hand_visible, action_segments, contact_events, and the wrist IMU and camera streams — but each format projects them differently. RLDS (TFDS-compatible tfrecord shards) and LeRobot both fold hand pose and wrist signal into the per-step observation and action structure their respective loaders expect; LeRobot’s feature set explicitly includes observation.images.wrist_l/r alongside observation.state and action. HDF5 and zarr deliver the fields closest to their native form: one group per episode, time-major chunking, and per-stream nanosecond timestamps, which is why Firsthand’s own QA tooling reads that format — it is the most complete for verifying sync directly.

WebDataset is the exception in structure rather than content: it keys samples by frame rather than by episode, with hand-pose and action annotations delivered inline as ann.json alongside the image and depth keys for that sample. Whichever format is chosen, the canonical field layout is /formats/episode-schema — every delivery format is a projection of it, so switching formats later does not change what a hand-pose or wrist label actually contains.

A spec for action-labeled egocentric pretraining data should state the wrist and hand-pose fields explicitly rather than leaving them implied: which hand-pose fields are required (hand_joints and hand_visible, at minimum, for any dexterous task), whether wrist IMU, wrist RGB, or both are needed, whether action_segments should carry verb/noun labels for specific skills, and the calibration and sync tolerances the delivery must meet. /custom-collection/action-labeled-egocentric-video and /custom-collection/pretraining-data-collection both start from this same field set and adapt scope, task diversity, and volume to the target embodiment.

To request a custom action-labeled pretraining collection with a specific wrist and hand-pose field spec, start at /custom-collection/pretraining-data-collection or /custom-collection/action-labeled-egocentric-video.