Can Egocentric Human Video Train VLA Models?

Partly, and only as one part of a larger recipe. Of the six papers cited on this page, two describe using human video at all. EgoScale (https://arxiv.org/abs/2602.16710) trains a vision-language-action (VLA) model on over 20,854 hours of action-labeled egocentric human video, reports a log-linear scaling law between human data scale and validation loss, and says that validation loss strongly correlates with downstream real-robot performance. GR00T N1 (https://arxiv.org/abs/2503.14734) places human videos at the base of a data pyramid and gets actions for them from a latent-action codebook and a trained inverse dynamics model.

Neither paper says human video is enough by itself. EgoScale pairs large-scale human pretraining with a lightweight stage of aligned human-robot mid-training and then post-trains on task-specific robot demonstrations. Its introduction says that, rather than replacing robot data, large-scale human demonstrations amplify its effectiveness. GR00T N1 mixes human video with real-robot trajectories and synthetic data. So the honest answer to the query is that human video can supply a large share of the training signal in some recipes, with robot-side data still doing work in the same recipes.

This guide reports what each paper says about its own data and its own robot. It does not treat any finding as universal, and it makes no claim that Firsthand data trained, fine-tuned or evaluated any named model. The sections below cover which papers use human video, at which stage, how they handle the embodiment gap, which capture and label fields make first-person video useful, and where the approach runs out.

The table lists each paper once. Where the abstract, README or text read for this page does not describe human video, the row says Not stated. That records what was read, not a claim that the paper never used any.

Octo is described by its paper as a generalist robot policy and is not called a VLA here. Four of the six papers in the table describe robot-side pretraining data in the parts read, which is why this guide does not present human video as the default way to train a VLA.

The two papers place it differently. EgoScale describes a three-stage pipeline. Stage I is human pretraining on a mixture of egocentric human activity datasets totaling 20,854 hours. The paper says the majority are in-the-wild egocentric recordings across household, industrial, retail and educational settings, covering 9,869 scenes, 6,015 tasks and 43,237 objects. The recordings are RGB at 30 FPS, and camera motion and hand trajectories are recovered with off-the-shelf SLAM and hand-pose estimation. The paper says those estimates are noisy because the collection is unconstrained, and argues that the scale and diversity of the data still provide effective supervision.

Stage II is aligned human-robot mid-training. The paper describes a smaller dataset of 344 tabletop manipulation tasks, each captured in roughly 30 human trajectories and 5 robot trajectories, about 50 hours of human data and only 4 hours of robot data. Human demonstrations in that set use the same camera configuration as the robot, with matched viewpoints and calibrated intrinsics. Stage III post-trains on task-specific robot demonstrations. The paper says the two data stages decouple scale from alignment: Stage I provides diversity and semantic grounding, while Stage II supplies precise human-robot correspondence.

GR00T N1 uses human video earlier and more loosely. In its data pyramid, large quantities of web data and human videos form the base, synthetic data from simulation and generated video sits in the middle, and real-robot data sits at the top. The paper says it learns across the pyramid in both pre-training and post-training. Because human videos carry no actions, it annotates them with latent actions and inverse-dynamics pseudo-actions so they can be treated as additional embodiments.

For a data team the difference matters. An EgoScale-style recipe needs action-labeled human video at scale plus a small aligned human-robot set. A GR00T N1-style recipe can take action-less video, with the action labels inferred by models. The two recipes ask for different collection specs, and neither paper describes the other recipe.

The embodiment gap is the mismatch between a human body and a robot. EgoScale names two core challenges in its method section: human demonstrations are noisy and lack paired robot actions, and human and robot embodiments differ substantially in kinematics and control interfaces. Its introduction adds that many prior human-to-robot transfer efforts targeted grippers or low-DoF hands, where fine-grained finger articulation is absent, leaving open whether human data can meaningfully support complex, dexterous manipulation at scale. The table below lists each problem a paper names and how that paper responds.

EgoScale also reports that although human actions are supervised in a high-DoF dexterous hand space, the learned representations transfer to robots with lower-DoF hands, and it describes this as an embodiment-agnostic motor prior. That is a result on its own data and robots. The Firsthand site states that it captures first-person human video, not robot or teleoperation data, and that using the episodes for a specific robot means your own pipeline bridges the embodiment gap. So the mapping from human hand signals to a robot action space stays a decision in your written spec.

Egocentric means the camera moves with the person doing the task, so hands, tools and the object being handled stay in frame. EgoScale builds on that. Each human demonstration in its pretraining data consists of egocentric RGB video, with camera motion and hand trajectories recovered by off-the-shelf pipelines. It then expresses arm motion as relative wrist motion between timesteps, which removes dependence on absolute camera pose and global camera movement.

The Firsthand site makes a related argument from the robot side: a robot or humanoid perceives from a head- or wrist-mounted camera, so egocentric capture matches that viewpoint and the policy trains on the geometry it will see at inference. Where depth matters, the site describes calibrated head-mounted stereo capture in which every frame pair carries recoverable depth, and says stereo, depth and IMU streams are hardware-triggered together on one clock. Neither the paper nor the site claims that viewpoint alone closes the embodiment gap.

Raw first-person footage tells a model what a scene looked like. The labels tell it what the hands did. EgoScale supervises on wrist motion and hand articulation, and its model conditions on an image and a language instruction. The table lists the capture and label fields the Firsthand site states, what each gives VLA training, and where it sits in the delivered episode. Field names are given only where the site states them.

On language, the site states one task instruction per episode and nothing more. Narration, captions, dense language annotation, subtask language and reasoning traces are not stated as Firsthand deliverables. Episodes are delivered in RLDS, LeRobot, WebDataset or HDF5.

The papers state several limits, and they point the same way. Human video lacks robot actions, so the action signal is estimated: EgoScale recovers hand trajectories with off-the-shelf pipelines that it calls noisy, and GR00T N1 infers pseudo-actions with a trained model. Human and robot bodies differ, so both recipes add an alignment step. And the reported gains come from specific data on specific robots, such as EgoScale’s 54% average success improvement over a no-pretraining baseline using a 22-DoF dexterous robotic hand. That figure describes that setup, not every model or task.

The practical reading: human video is a candidate source for the broad, cheap-to-scale part of training, and the papers that use it still add robot-aligned data afterwards. Whether it helps your model is something to test against your own robot and tasks, and a pilot batch is a cheaper way to find out than a full collection.

Firsthand collects custom egocentric data fast and on demand, to a written spec, for teams that need a capture a public mixture does not cover. Each episode carries egocentric video, 3D hand pose with a per-joint visibility flag, wrist trajectories, contact events, action segments and one task instruction, and is delivered in RLDS, LeRobot, WebDataset or HDF5 under a perpetual, buyer-owned license. The VLA pretraining page describes the broad, diverse capture a pretraining stage draws on, and the fine-tuning page describes the narrow task set used later.

• Name the stage first. Broad human-side data for pretraining and a narrow aligned set for mid-training or fine-tuning are specified differently.
• Decide how hand signals map to your robot. Firsthand captures first-person human video, not robot or teleoperation data, so your own pipeline bridges the embodiment gap and the mapping belongs in the written spec before capture.
• Choose the labels you need from the fields above, and mark anything you cannot get from human capture, such as robot proprioception, as coming from your own pipeline.
• Keep the language request exact: one task instruction per episode.
• Pick the delivery format your training code already reads, request a sample, and check it against your pipeline before scaling the collection.

Moving fast on data comes from settling the spec early, not skipping it. The custom dataset request starts that written spec.