Humanoid Robot Training Data Sources Compared

Humanoid robot training data comes from four main sources. Robot teleoperation records a person driving the robot itself. Egocentric human video records a person doing a task from a head-worn viewpoint. Simulation and synthetic data generate robot trajectories or videos with software. Motion capture, marker-based or markerless, measures human body motion inside an instrumented volume, typically a studio or lab. This page describes teleoperation, simulation and motion capture only generically: no vendor, accuracy, frame rate or price.

The NVIDIA GR00T N1 paper (arXiv 2503.14734) gives the standard picture. It arranges its training corpus as a data pyramid: web data and human videos at the base, synthetic data from physics simulation or neural models in the middle, and real-world data collected on physical robot hardware at the top. Data quantity decreases and embodiment specificity increases moving up. The paper also states that no internet-scale humanoid dataset exists and that real-world humanoid data are costly and time-consuming to acquire at scale. See /glossary/data-pyramid.

Firsthand fills one box. It collects custom egocentric human video on demand, to a written skill spec, and the site states that it does not capture robot or teleoperation data. Firsthand claims no relationship with the GR00T N1, EgoScale, EgoVerse or HumanScale authors or any lab cited here. For a by-stage view of the same sources, see /guides/humanoid-robot-data-collection-methods; this page compares them side by side across setting, scale, cost drivers and formats, and adds motion capture.

The first table sets the four side by side. Teleoperation logs the robot’s own sensor streams and controls while an operator drives it. GR00T N1’s teleoperation, for example, captured an operator’s wrist poses and finger movements with trackers and gloves, retargeted them to humanoid actions with inverse kinematics, and recorded head-camera images at each step. That is one paper’s setup, not a rule.

Simulation produces robot trajectories in a physics simulator; GR00T N1 expands a small set of human demonstrations this way. Neural generation produces videos with no action labels, so the paper labels them with latent actions or inverse-dynamics pseudo-actions (see /glossary/latent-actions). Motion capture records body or marker motion over time; video, depth and contact streams need separate systems.

Firsthand records what the wearer sees and does: a global-shutter head camera at 4K and 60 fps, two wrist cameras at 1920×1080 and 60 fps, depth at 848×480 and 30 fps recovered from a calibrated stereo pair, and 6-DoF IMUs on the head and both wrists at 200 Hz. Ten streams are synchronized on one PTP clock, with a median skew of 1.12 ms. The second table maps each delivered field to a humanoid-learning use and its field name.

Teleoperation has the smallest gap for the robot it was collected on, because the actions are in that robot’s own space. The cost is portability: GR00T N1 describes pooled robot data as an archipelago of “data islands” because embodiments, sensors and control modes vary.

Egocentric human video and motion capture share the same gap: the data shows a human body, not a robot. HumanScale (arXiv 2606.20521) calls the embodiment gap egocentric video’s main weakness and says a smaller amount of kinematically aligned robot data in post-training is what corrects it. EgoScale (arXiv 2602.16710) supervises on relative wrist motion and retargeted hand articulation. Firsthand delivers measured pose and IMU layers, not retargeted robot actions, and no retargeting step is stated as delivered. See /glossary/embodiment-gap and /glossary/motion-retargeting.

Simulation can match the robot exactly, but the gap moves to appearance and physics. GR00T N1 notes that using simulation datasets can be challenging because of the simulation-to-reality gap, and that existing synthetic generation still struggles with diverse, counterfactual data that obeys physics. See /glossary/sim-to-real-gap. No source linked here puts a number on any of these gaps, so this page does not either.

Teleoperation is the better choice when the policy must run on one specific robot and you need ground-truth actions in that robot’s action space with no retargeting. GR00T N1 places real-robot data at the top of its pyramid to ground the model in real-robot execution. HumanScale makes the same split: post-training emphasises alignment, and teleoperation data is strongest on embodiment alignment.

It is the source that records the robot’s own sensing and control, which human video and motion capture do not.

The costs are recurring. GR00T N1 states that robot data scales linearly with human labor, and HumanScale lists a robot, a teleoperation rig, a trained operator, a staged scene and ongoing maintenance. No price is stated here. Firsthand does not capture teleoperation data; see /compare/human-demonstrations-vs-teleoperation for how human demonstrations and teleoperation relate.

Simulation and synthetic data help when you need to multiply a small set of demonstrations or cover scenes that are slow to stage. GR00T N1 reports generating 780,000 simulation trajectories, which it equates to 6,500 hours of human demonstration, in 11 hours, and about 827 hours of neural-generated video from 88 hours of its own teleoperation data. Those are that paper’s figures for its own pipeline and compute, not Firsthand figures.

The limits are the ones above: a sim-to-real gap and physics fidelity. The paper also notes that liquids and articulated objects are extremely challenging to simulate. GR00T N1 uses synthetic data alongside real-robot data and human video, not in place of them.

Firsthand states no simulation or synthetic data offer. See /compare/egocentric-vs-synthetic-data for how the site frames real-environment capture against synthetic data.

Egocentric human video scales better when the dataset must span many tasks and real environments. HumanScale compared egocentric and teleoperated real-robot pretraining at a matched 5,000 hours and reports that the egocentric model generalised better to unseen tasks, while calling its results preliminary. EgoScale reports a log-linear relationship between egocentric pretraining hours (20,854 in its study) and validation loss, and says large-scale human demonstrations amplify robot data rather than replace it.

For humanoids the fit is structural. A humanoid with a head-mounted camera occupies nearly the viewpoint a head-worn rig records, two wrist cameras keep both hands in frame, and 24-joint body pose carries stance and reach. See /custom-collection/humanoid-robots. The site states no throughput or hour count, so scale here means capture wherever the task is done, not a rate.

Motion capture is the better fit when you need controlled, whole-body measurement of a person in a fixed volume, or a measurement reference. EgoScale itself used a tracker-and-glove motion-capture stack to record its aligned human data. The limits are the setting and the missing task context: participants come to the volume, and it records only what the studio is arranged to capture. See /compare/egocentric-video-vs-motion-capture.

The site does not claim teleoperation or any robot-in-the-loop collection, simulation or synthetic data, or motion capture studio capture. It does not claim robot-frame action labels or a retargeting step, a head or camera 6-DoF pose, or a SLAM trajectory. A 6-DoF head IMU at 200 Hz is delivered, but that is inertial data, not a camera pose.

There is also no third-person capture, no narration or dense captions (one task instruction is paired to each episode), no evaluation or benchmark datasets, no hand or body pose accuracy figure, and no capture at a customer’s own facility. The fourth table lists each item. A “not stated” row means no page on this site says it is delivered, not that it is impossible; raise it in the skill spec before the quote.

Most programs blend sources, and the papers describe how. GR00T N1 co-trains across its whole pyramid and treats action-less human video, once labelled, as an additional embodiment. EgoScale runs two stages: large-scale egocentric pretraining, then a lightweight mid-training on aligned human and robot data, about 50 hours of human data and 4 hours of robot data in its study, recorded with the robot’s own camera configuration. HumanScale’s recipe has the same shape: pretrain on egocentric video, then adapt with a small amount of labelled robot data.

EgoVerse (arXiv 2604.07607) adds a caution. Its multi-lab study found that co-training with human data helps, but that scaling benefits depend on aligned human–robot data that share task semantics and scene context. The third table summarises what each linked source states and where it is silent.

Practically, keep the sources separable. Record each episode’s source in your training manifest, keep each source’s action representation distinct, and agree shared tasks and start and stop cues in the skill spec so aligned human and robot sets are comparable. Firsthand states no alignment method with an outside robot system and delivers no robot-side data; see /custom-collection/humanoid-pretraining-data.

Send the humanoid tasks, the environments and regions, the capture tier and annotation depth, and the accept criteria. The order flow is spec, quote, Spec Pilot, then production. RGB is the baseline and other streams are added where the spec calls for them. Hand pose, segmentation and action labels each add equipment, QA passes and cost, so order what your model consumes. There is no list price; billing is per validated hour.

Where coverage is deep, the median time from a signed spec to a first validated batch is 48 hours; where coverage is thin it is typically three to four weeks. On-demand top-ups run against the same spec after each training run.

Formats are RLDS, LeRobot, WebDataset, HDF5 / zarr and Rerun .rrd. In the LeRobot v2.1 export, hand and body pose are flattened into one observation.state vector, 2×21×3 hand joints first and then 24×3 body joints. Every episode is checked on sync, calibration, coverage and consent; a rejected episode is recaptured at the provider’s cost; faces, plates and screens are blurred before delivery. Delivered data ships under a perpetual, buyer-owned license, and exclusivity is available.