How Does Human-to-Robot Transfer Work in Robot Learning?

Human-to-robot transfer is the process of training a robot policy using skills learned from human demonstration — most often egocentric human video — rather than, or in addition to, data collected directly from the robot itself. The premise is that a human performing a manipulation task and a robot performing the same task share enough underlying structure — how a reach unfolds, when contact happens, how a grasp closes — that a model can learn useful priors from the human recording and carry some of that learning onto the robot.

Transfer is not automatic or free. A human body and a robot’s hardware differ in degrees of freedom, grasp geometry, reach, and camera viewpoint — a mismatch documented in full in what is the embodiment gap. Human-to-robot transfer is the broader question of how a training pipeline bridges that mismatch so the policy actually benefits from human data rather than being confused by it.

This page explains the mechanism of transfer. For the egocentric capture methods referenced throughout — viewpoint matching, calibration, sync, and labels — see how to collect robot pretraining data and how to collect aligned human-robot mid-training data.

A robot cannot execute what it sees in human video by directly imitating pixels, because the body performing the action in the footage is not the body that has to execute it. The embodiment gap is the mismatch between a human’s hands, arms, and sensing and a robot’s hardware — a parallel-jaw gripper instead of five fingers, different arm kinematics, a different camera mounted in a different place. Human demonstration data carries that gap by construction: it was recorded from a human’s body, not the target robot’s, so a retargeting or alignment step has to sit between the recording and anything a robot policy can act on.

This is also why human video on its own is never treated as a complete substitute for robot data. It scales breadth — coverage across tasks, sites, and people — far faster than a robot-and-operator fleet can reach, which is exactly why large-scale pretraining leans on it. But closing the embodiment gap itself requires real robot data layered on top, at the mid-training and post-training stages, aligned against the human-trained prior rather than replacing it.

EgoScale’s published result is the clearest public evidence that this gap is solvable at scale: a small, aligned mid-training and post-training stage on real robot data, layered on top of large-scale egocentric pretraining, measurably lifts real-robot task success rather than leaving the pretrained prior stranded on the human side of the gap (arXiv 2602.16710; NVIDIA GEAR Lab project page).

A policy does not learn directly from raw video pixels. It learns from a representation of what changed — how a hand moved, when contact happened, what the hand was doing — extracted from the footage and expressed in a form a model can train on. Three representations show up across published work, and they are not mutually exclusive.

• Hand-pose and action-segment labels — 3D hand-pose keypoints and action_segments records ({start_ns, end_ns, verb, noun}) captured or derived directly from the footage, timestamped against a shared clock so a segment boundary lines up with the exact frame where a sub-action starts or ends.
• Latent actions — a learned, compressed representation of what changed between two video frames, inferred by a model directly from the footage without ever being told what the underlying action actually was. NVIDIA’s GR00T N1 paper (arXiv 2503.14734) uses latent actions to let unlabeled egocentric human video contribute action-relevant training signal even though it carries no explicit robot action label.
• Inverse-dynamics pseudo-labels — action labels inferred after the fact from off-the-shelf SLAM and hand-pose estimation run on video that was not captured with a robot action recorded alongside it. EgoScale’s pretraining corpus is labeled this way: action-relevant signal is derived from the 2D or monocular footage, not measured directly against a synchronized depth stream and hardware clock at capture time.

The tradeoff between these approaches is provenance and precision, not usability — EgoScale’s own published results demonstrate that pseudo-labeled action signal is good enough to produce a real scaling trend and a real downstream success-rate gain. What pseudo-labels cannot offer is the same geometric grounding as a label measured directly against synchronized depth and a shared hardware clock at the moment of capture, which is why a live-captured hand-pose and action-segment label carries more direct supervisory signal than one inferred from appearance alone.

GR00T N1’s data pyramid places these representations at different layers by abundance and specificity: web-scale data and human video sit at the wide base, where latent actions and inverse-dynamics pseudo-labeling let unlabeled footage still contribute signal; synthetic data sits in the middle; and real robot data with directly measured actions sits at the narrow top, where embodiment alignment matters most.

Human-to-robot transfer happens across three stages, not in one step. Pretraining teaches broad visual, physical, and action priors from large-scale data — often egocentric human video — without needing to come from the target robot at all. Mid-training is a much smaller, embodiment-matched dataset that aligns those priors to one target robot’s camera viewpoint, calibration, and task set, pairing human and robot capture on the same pipeline. Post-training (fine-tuning) narrows further to the exact deployed task, environment, and success criteria. This same structure is documented stage by stage in pretraining vs mid-training vs post-training robot data.

EgoScale’s published recipe is the clearest public illustration of how the stages connect: 20,854 hours of action-labeled egocentric human video at pretraining, followed by a lightweight mid-training stage pairing roughly 50 hours of human video with about 4 hours of robot data across 344 tabletop tasks. The result reported is a 54% average success improvement on a 22-degree-of-freedom dexterous hand versus skipping pretraining entirely, alongside a log-linear scaling relationship between pretraining data volume and validation loss that correlates with real-robot downstream performance (arXiv 2602.16710; NVIDIA GEAR Lab project page).

The ratio between stages, not a specific hour count, is the transferable lesson: pretraining is measured in tens of thousands of hours because breadth matters most there; mid-training is measured in dozens of hours because precision on one embodiment matters most there; post-training is scoped to the exact deployment task and its edge cases. A program targeting a different robot or task set will land on different hour counts at each stage, but the same three-stage shape — broad, then aligned, then narrow — is what the published EgoScale numbers demonstrate.

EgoScale is the most direct published study of human-to-robot transfer end to end: a two-stage recipe — large-scale egocentric pretraining followed by a lightweight, aligned mid-training stage — that reports a 54% average real-robot success improvement on a 22-degree-of-freedom dexterous hand, plus a log-linear scaling law between pretraining volume and validation loss (arXiv 2602.16710; NVIDIA GEAR Lab project page).

NVIDIA’s GR00T N1 paper (arXiv 2503.14734) describes the mechanism for mixing data sources with and without explicit robot actions — latent actions and inverse-dynamics pseudo-labeling — arranged in a pyramid from broad web and human video at the base to real robot trajectories at the top, which is the structural model the transfer pipeline in this guide follows.

Generalist’s GEN-0 program frames pretraining-stage transfer around diversity, stating explicitly that data quality and diversity matter more than sheer volume — a framing that applies one stage earlier than EgoScale’s mid-training result but points at the same underlying principle: how well data is matched to what a model needs to learn determines transfer more than raw hours do (GEN-0 blog post).

HumanScale (arXiv 2606.20521) and Figure’s Project Go-Big are both pursuing large-scale human data collection for the same pretraining-to-transfer purpose, and Figure’s official project page states a commitment to scaling humanoid data collection as part of training its humanoid robots — evidence that leading humanoid developers treat human-to-robot transfer as a primary lever worth investing in deliberately, not a side effect of simply collecting more robot data.

Viewpoint is the first lever: footage captured from the same first-person framing the robot’s own cameras will see at inference transfers more directly than third-person footage, because the training distribution already matches deployment rather than needing to be reinterpreted across a viewpoint change.

Calibration is the second. Every camera needs its own intrinsic solve — focal length, optical center, and lens distortion coefficients (fx, fy, cx, cy, k1, k2, p1, p2, k3) — plus an extrinsic transform placing it relative to the others, gated on a reprojection-error threshold (rms_reproj_px < 0.28) before its frame-level labels are trustworthy. A hand-pose or action label built on uncalibrated geometry cannot carry accurate transfer signal no matter how well the rest of the pipeline is built.

Cross-stream synchronization is the third: every stream — video, depth, hand pose, and action_segments — hardware-triggered onto a single PTP clock domain, so the relationship between what a hand did and when contact happened survives into training rather than being reconstructed approximately after the fact. At the mid-training stage specifically, this synchronization is what makes a human episode and a robot episode on the same task directly comparable, frame for frame.

Label quality is the fourth lever, and it is the dimension EgoScale’s own results show as a tradeoff rather than a solved problem: a live-captured hand-pose and action-segment label measured against synchronized depth at capture time carries more direct supervisory signal than an inverse-dynamics pseudo-label inferred from appearance after the fact. Diversity across tasks, scenes, objects, and demonstrators is the fifth, and it is the dimension GEN-0’s published framing names explicitly as mattering more than raw volume — a pretraining corpus with narrow coverage transfers worse than a smaller, more diverse one, regardless of hour count.

Transfer from human video alone does not close the embodiment gap; it narrows what the mid-training and post-training stages still have to do. A pretrained-only model has never seen the target robot’s actual degrees of freedom, grasp geometry, or camera rig, so its priors are general rather than deployment-ready — which is exactly why EgoScale’s recipe still needs a real-robot mid-training stage on top of its pretraining corpus, not pretraining alone.

Pseudo-labeled action signal — inverse dynamics and latent actions applied to unlabeled video — is inferred from appearance, not measured against synchronized depth and a hardware clock, so it carries less geometric precision than a label captured live. That is a limitation of provenance, not of whether pseudo-labeling is usable; EgoScale’s results show it works well enough to produce a real scaling trend and a real success-rate gain, but a program that needs tighter geometric grounding should weigh that tradeoff against directly captured labels.

There is no single published figure for how much mid-training or post-training data is "enough," because the right amount depends entirely on how many distinct tasks, environments, and edge cases the deployed model must cover, and how much of that coverage pretraining and mid-training already established. This is why mid-training and post-training programs are typically scoped through a small pilot batch validated against a written spec, rather than a fixed target set in advance.

Because transfer quality depends on viewpoint match, calibration, sync, and label quality — not just hours — it is a poor fit for whatever egocentric footage happens to already exist. A static corpus was not captured with any particular robot’s camera geometry or task list in mind, so it transfers only as well as it happens to resemble the target embodiment by coincidence. On-demand collection to a written spec is the alternative: the spec names the viewpoint, calibration targets, task list, and label schema the transfer pipeline needs, and capture is run against that spec rather than filtered out of existing footage after the fact.

Firsthand captures the egocentric video, depth, 3D hand pose, and action-segment data used as the human side of a pretrain-then-align pipeline — collected fast, on demand, and to a written spec, then delivered in RLDS, LeRobot format, or WebDataset with a buyer-owned commercial license. Every stream is hardware-triggered onto one PTP clock domain and calibrated per episode, so hand-pose and action-segment labels carry the geometric grounding a transfer pipeline depends on rather than being inferred after the fact. For the aligned, embodiment-matched slice that closes the gap between pretraining and a specific robot, see how to collect aligned human-robot mid-training data.

Firsthand does not capture robot-embodiment action data itself; the mid-training and post-training data that completes transfer onto a specific robot comes from a program’s own robot-and-operator or teleoperation pipeline, run against the same written spec and aligned to the egocentric pretraining corpus.