Bimanual Manipulation Pretraining Data

Bimanual manipulation pretraining data is egocentric human video of two-handed tasks captured broadly across objects, environments, and coordination patterns, with per-hand 3D pose, contact events, and depth, rather than deep repetition of one specific skill.

Where fine-tuning data narrows in on the exact tasks a policy must execute, pretraining data is meant to teach general two-hand coordination first: how hands hand off, stabilize for each other, and sequence actions across a wide range of settings, before any task-specific narrowing happens.

A robot that only ever sees demonstrations for the exact tasks it will perform has no fallback when conditions shift slightly a different object shape, a new grip point, a partial occlusion. Broad pretraining on human two-hand behavior gives a model a general prior for how two hands coordinate, which fine-tuning on task-specific data can then specialize.

This mirrors the broader pretraining logic already established for robot foundation models: NVIDIA’s EgoScale research demonstrated a log-linear scaling law between the scale of egocentric human pretraining data and validation loss, and GR00T N1’s data pyramid places web data and human video at the base precisely because that broad layer establishes priors that narrower robot data later refines.

Each episode needs 3D hand pose tracked separately for each hand, with visibility flags for the frequent occlusion two-hand work produces one hand crossing in front of the other, or in front of the object. Contact events mark exactly when and where each hand makes and breaks contact, aligned to the same clock as the video and depth streams.

Wrist trajectories matter as much as hand pose: the continuous path a wrist follows through a handoff or a stabilizing hold carries coordination information that a single start-and-end pose loses. Capturing both hands’ wrist motion continuously, not just at key frames, is what lets a pretraining corpus teach coordination rather than just endpoints.

A synchronized depth stream rounds out the episode, giving each hand’s pose a grounded position relative to the object and to the other hand, and instance segmentation and action segments give the raw pose and contact stream the structure a pretraining pipeline needs to slice episodes into learnable units.

Per-hand pose is only useful if it is aligned to video and depth on the same clock every time and calibration drifts between cameras corrupt exactly the signal a two-hand coordination model most needs: the relative timing between what each hand is doing. Firsthand aligns all ten streams under 2 ms and validates calibration drift as part of per-batch QA.

Occlusion is common in two-hand work one hand passes in front of the other, or both hands disappear briefly behind the manipulated object so visibility flags on hand pose matter more here than in single-hand capture. A tracker that silently guesses through occlusion introduces noise a pretraining corpus cannot afford at scale.

A pretraining spec for bimanual skills should vary coordination pattern (handoff, simultaneous, sequential, one-hand-stabilizes) independently from object type and environment, so the resulting corpus does not accidentally correlate a coordination style with a specific object shape. Grasp diversity per hand asymmetric grasps, power versus precision grips on each side should be planned the same way single-hand dexterous pretraining plans grasp diversity.

Environment and lighting variation follow the same diversity logic already used across Firsthand’s custom collections: breadth is planned and specified up front, not left to whatever a single capture session happens to produce, because a pretraining corpus that is deep in one setting and thin in others teaches a biased prior.

It also helps to plan for the handedness of contributors and the relative roles each hand plays across a task set, since a corpus that only ever shows a dominant hand leading and a non-dominant hand stabilizing teaches that pattern as if it were universal. Rotating which hand leads across otherwise similar tasks, and specifying that rotation in the written spec up front, keeps the coordination prior from encoding a handedness bias the deployed policy would then inherit.

Bimanual pretraining data, described on this page, is broad: many objects, environments, and coordination patterns, sized to teach a general two-hand coordination prior. Bimanual fine-tuning data the angle covered on our Custom Bimanual Manipulation Data Collection page is narrow and deep: the exact two-handed tasks, objects, and environments a specific policy must execute, collected against tight accept criteria for that use case.

EgoScale’s published two-stage recipe illustrates the shape of this split at a different skill level: a large pretraining corpus (20,854 hours) followed by a comparatively small, tightly aligned mid-training stage (roughly 50 human hours plus about 4 robot hours across 344 tabletop tasks). Bimanual programs follow the same shape breadth first, narrow fine-tuning second even though Firsthand has not published hour counts specific to bimanual capture.

Humanoid platforms are bimanual by construction, so a general two-hand coordination prior is directly relevant to nearly every manipulation skill a humanoid policy will eventually need, not just a subset of tasks. Dual-arm industrial and lab platforms benefit the same way when their task set spans multiple coordination patterns rather than one fixed motion.

Both team types typically pair this pretraining-scale data with a narrower, task-specific fine-tuning collection once the target skills are defined, following the same pretraining-then-fine-tuning structure the site documents for dexterous hand and other manipulation pretraining.

The embodiment gap between a human demonstrator and a robot’s own kinematics is real, which is why a broad pretraining stage on human video is typically followed by an aligned mid-training or fine-tuning stage on the robot’s own embodiment rather than deploying directly from human data. For bimanual platforms specifically, that alignment stage matters even more, because the relative geometry between two arms varies far more across robot designs than a single arm’s kinematics does.

Describe the coordination patterns, object categories, and environment diversity your pretraining corpus needs. We turn it into a written spec covering capture tier, annotation depth, and accept criteria, then quote reach, timeline, and price against that document before capture begins.

The collection runs through the same seven-stage end-to-end pipeline as every Firsthand program: contributor recruiting, capture, per-batch QA, anonymization, and licensed delivery. Where coverage is already deep, a first validated batch ships in a median of 48 hours; thin or unusual environments typically take three to four weeks because vetting operators there is the bottleneck, and we quote that up front.

A short pilot batch is the usual starting point: it lets your team validate hand-pose accuracy, sync quality, and label format against your own pipeline before committing to the full pretraining volume the spec calls for.