Humanoid VLA Training Data

A vision-language-action (VLA) model takes camera images and a language instruction and outputs actions. The GR00T N1 paper describes that split for a humanoid: a vision-language module interprets the environment through vision and language instructions, and a diffusion transformer module generates the motor actions. Humanoid VLA training data is whatever supplies those three legs for a humanoid target: observations, an instruction, and an action signal.

Firsthand supplies one slice of that. It captures first-person human demonstration episodes against a written spec, with head-camera video, wrist cameras, depth, IMU and measured hand and body pose. It is not a complete humanoid corpus. Robot trajectories, a humanoid state vector and an action space for your embodiment come from your own stack, and nothing here implies that any named model was trained or evaluated on Firsthand data.

Several neighbouring pages cover related ground, so the comparison table below shows which question each one answers. This page takes the VLA-model view: what a humanoid VLA team needs to specify, what the human slice contains, and what the team has to bring.

The GR00T N1 paper states the data problem plainly: real-world humanoid data are costly and time-consuming to acquire at scale. It responds by organizing its training corpora as a data pyramid, with web data and human videos at the base and real-world data collected on physical robot hardware at the top. It describes its human video datasets as having no explicit action labels while providing detailed first-person perspectives of hand-object interactions, and it describes learning latent actions for those videos in place of ground-truth actions.

The EgoScale paper takes a different route. Its abstract describes training a VLA model on action-labeled egocentric human video, and a two-stage recipe of large-scale human pretraining followed by lightweight aligned human-robot mid-training. The authors read the transfer to robots with lower-DoF hands as evidence of a reusable, embodiment-agnostic motor prior. That is the authors’ interpretation of their own results, and it concerns a dexterous robotic hand rather than any particular humanoid.

For a humanoid team the practical points are viewpoint and labels. A head-worn rig records from a head-camera viewpoint, which is the viewpoint the humanoid-robots page frames as the fit for humanoid work, and measured hand pose and contact events turn video into an action-relevant signal. Both papers pair human video with robot data, so first-person human data is one input to a humanoid VLA program rather than a replacement for the rest of it. Egocentric video alone is not sufficient to train a humanoid VLA, and this page does not claim that it is.

Firsthand captures first-person human video, not robot or teleoperation data, and capture does not take place at a customer’s own facility. Using the episodes for a specific robot means your own pipeline bridges the embodiment gap. The site does not state a retargeting method, a humanoid joint mapping or an action space, so none is assumed here.

The GR00T N1 paper describes its teleoperated humanoid data differently. A human operator’s wrist poses and finger movements are tracked and retargeted to humanoid actions with inverse kinematics, and each step records head-mounted camera images alongside the robot’s actions. The same paper notes that robot data scales with human labor, because an operator is typically needed to produce each trajectory.

The table below sets the two side by side using only the site’s own statements and the GR00T N1 paper. Where neither source says, it reads Not stated.

Each attempt at a task is one episode. The site describes the wider capture stack as ten streams: a 4K head camera, two wrist cameras, depth, IMU, 3D hand and body pose, segmentation and action labels, hardware-aligned under 2 ms. Calibration is accepted only under a 0.28 px RMS reprojection gate.

The action-relevant fields are the ones the episode schema lists. Hand pose is 21 joints per hand, in 3D, with a per-joint visibility flag. Wrist trajectories, contact events for the moment a hand meets an object, and action segments with start and end timestamps, a verb and a noun sit alongside depth and 200 Hz 6-DoF IMU streams from the head and both wrists.

The only language field the site states for Firsthand capture is one task instruction per episode. Every action field is continuous, so chunking, tokenization or retargeting to a humanoid action space is a step your pipeline performs after delivery.

Tasks are whatever the written spec says. The humanoid-robots page frames the fit as bimanual and whole-body work recorded from a head-worn viewpoint, with body pose and IMU adding stance, reach and the transfer between locomotion and manipulation. Dedicated pages cover task families such as sorting, packing and kitting, box and parcel handling, grocery put-away, bimanual manipulation and mobile manipulation, and each states its own coverage.

The spec also fixes environments, regions, capture tier, annotation depth and accept criteria, and the quote for reach, timeline and price is written against that document before any capture begins. Humanoid-specific task lists, task durations and per-task episode counts are Not stated on the site, so they are set in the spec rather than assumed.

Every delivery format is a projection of one canonical episode schema with four groups: meta, streams, annotations and calibration. Timestamps are per-stream nanosecond values on one clock, and streams are stored at native rate, so resampling happens in your loader rather than in the delivery.

In the LeRobot export, hand and body pose are packed into one observation.state vector: 2 x 21 x 3 hand joints first, then 24 x 3 body joints. Converter source is readable, so you can retarget the schema to your own loader, which is where a humanoid action space gets defined.

Faces, plates and screens are detected and irreversibly blurred at ingest, and a reviewer confirms the result before anything ships. The anonymization record is delivered with the data. Each episode links to a signed contributor release through a documented consent chain.

Collections ship under a perpetual, buyer-owned commercial license. Exclusivity is available per skill line or per environment as a separate negotiated term, and it has to be agreed before capture.

Where coverage is already deep, the first validated batch lands in a median of 48 hours from signed spec. Thin environments typically take three to four weeks, because vetting operators there is the bottleneck, and that timeline is quoted up front. On-demand top-ups run against the same spec and accept criteria after each training run.

The site also states program milestones: a pilot reaches its first delivery in two to three weeks, and a standing program reaches full rate in four to six weeks, both assuming coverage is already in place. Billing is per validated hour, and rejected items are recaptured at our cost. The site states no episode count, hour count or throughput for humanoid collections.

Start from what your humanoid VLA needs to see and do. Write down the tasks, the environments and regions, the label fields you need from the episode schema, and the accept criteria. Decide separately how your pipeline will retarget human hand motion to your humanoid and which action space it will use, because that work is yours and the site does not state it.

We turn the brief into a written spec and quote reach, timeline and price against it. Many teams begin with a Spec Pilot, a small, fast first batch that proves the spec and the accept criteria before volume scales on demand.