Dexterous Hand Pretraining Data

Dexterous hand pretraining data sits at the base of a high-DoF robot hand foundation model’s training, the same role human video plays in NVIDIA’s GR00T N1 data pyramid: web data and human videos at the base because they are the most abundant and cheapest to capture at scale, synthetic data in the middle, and real robot trajectories at the top because they are the scarcest and most expensive.

For a dexterous hand specifically, that base layer is not generic egocentric footage — it is footage with 3D hand pose, contact events, and wrist trajectories carried through every frame, because those are the signals that let a foundation model learn how a hand actually manipulates objects, rather than just where it is in the scene.

A high-degree-of-freedom robot hand has far more actuated joints than a parallel gripper, and teleoperating that many joints across enough tasks to build a broad prior is slow and expensive. Human hands already perform that full range of dexterous manipulation constantly, so egocentric human video is the layer that can realistically be captured at pretraining scale before a smaller, targeted robot-data stage follows it.

This mirrors why egocentric human capture became the pretraining source for robot foundation models generally: a person wearing a capture rig can move through far more grasp types, objects, and task settings per hour than a teleoperated robot hand can, at a fraction of the hardware cost.

EgoScale reports a log-linear scaling law between the amount of human pretraining data and validation loss, a trend the paper states correlates with real-robot task performance downstream. On a 22-degree-of-freedom dexterous hand, the paper reports a 54% average improvement in success rate over training with no human pretraining at all.

That result comes from EgoScale’s full two-stage recipe, not human pretraining in isolation: a 20,854-hour pretraining corpus of action-labeled egocentric human video establishes the broad prior, and a lightweight, aligned mid-training stage — roughly 50 hours of human video plus about 4 hours of robot data across 344 tabletop tasks — carries that prior onto the 22-DoF hand where the 54% figure was measured.

3D hand pose is the core signal: finger and wrist position tracked through the full reach-grasp-adjust cycle, with visibility flags so occlusion does not silently corrupt the label. Contact events mark exactly when a grasp or release happens, and wrist trajectories give the coarser motion path the hand followed to get there.

Video alone under-specifies what happened during a manipulation. Hand pose, contact events, and action segments turn raw footage into a signal a policy can actually learn a dexterous-manipulation prior from, the same distinction that separates a corpus useful for pretraining from one that is just a large pile of unlabeled clips.

Manipulation-grade 3D hand pose is only as trustworthy as the calibration behind it. Camera intrinsics and extrinsics have to be measured and recorded per session so pose estimates are metrically accurate, not just visually plausible, and every stream — video, depth, hand pose, contact events — needs to share a single hardware clock so fused signals line up at the frame level.

When calibration or sync drifts across a large pretraining corpus, the error does not show up as an obvious visual glitch; it shows up as a hand-pose or contact label that is subtly wrong in exactly the way a scaling-law study would need to be right to hold. That is why sync and calibration checks run on every batch, not just spot-checked samples.

A hand pretraining corpus is judged on breadth first: how many distinct grasp types (power grasps, precision pinches, in-hand reorientation), objects, and task settings it spans, because that breadth is what a later mid-training or fine-tuning stage builds on. A written spec sets that diversity target up front — which grasp families, which object classes, which environments — rather than letting volume alone stand in for coverage.

Generalist’s own framing from GEN-0 applies directly here: data quality and diversity matter more than sheer volume. A large pile of the same grasp repeated across similar objects adds hours without adding the coverage a dexterous policy actually needs.

Dexterous pretraining data is broad: many grasp types, objects, and settings, captured to build a general prior before a model is aligned to any specific hand or task. Dexterous fine-tuning (or mid-training) data is narrow: a smaller, closely scoped batch aligned to the target hand’s exact task set, captured on the same synchronized stream stack and accept criteria.

EgoScale’s recipe illustrates the ratio between the two: a pretraining corpus tens of thousands of hours deep is followed by a mid-training stage measured in dozens of hours, not thousands. Mid-training does not need to rival pretraining in scale; it needs to be closely matched to the target hand and camera geometry so the pretrained prior transfers cleanly.

This is the same overlap addressed on our general dexterous manipulation page: that page covers capturing fine-grained manipulation data for a specific policy’s task set, while this page covers the broader, pretraining-scale angle — building the general hand-manipulation prior a foundation model needs before any one task is specified.

The same ten-stream capture stack used across every Firsthand program, scoped broadly for pretraining rather than to one task: 4K head camera, two wrist cameras, depth, IMU, 3D hand and body pose, instance segmentation, and action labels, hardware-aligned under 2 ms.

Coverage is set by a written spec across grasp types, objects, and environments, so a dexterous hand pretraining corpus can span the diversity a foundation model needs rather than one recorded location. Because hand pose, contact events, and action labels are captured live on the rig rather than estimated from raw video afterward, every episode carries the same manipulation-grade signal a mid-training or fine-tuning batch would, at pretraining scale.

Every custom collection ships under a perpetual, buyer-owned commercial license, with exclusivity available so the data is never resold, and each episode links to a signed contributor release through a documented consent chain — the piece many public hand-pose corpora do not clear for commercial use.

Faces, plates, and screens are detected and irreversibly blurred at ingest, and a reviewer confirms the result before anything ships. The anonymization record is delivered with the data.

Start with the grasp types, objects, and task settings your current corpus is missing, rather than a total hour target. That gap becomes a written spec — tasks, environments, capture tier, annotation depth, and accept criteria — quoted for reach, timeline, and price before any capture begins.

Many programs start with a Spec Pilot, a small, fast first batch that proves the spec and accept criteria before scaling volume. From there the collection runs through the same seven-stage pipeline as every Firsthand program, with on-demand top-ups available afterward as new gaps in coverage show up during training.