Pretraining Data for Vision-Language-Action Models

A vision-language-action model takes a visual scene and a language instruction and outputs an action. During pretraining, that mapping is learned broadly, across many tasks, scenes, and objects, so the model builds a general prior before any single task is taught. That takes egocentric video, action labels (hand pose, contact events, action segments), and language annotation, captured across enough diversity that the model does not overfit to one narrow setting.

This is a different requirement from fine-tuning, where a small, task-specific batch is used to specialize an already-pretrained model. Pretraining data is judged on breadth and label quality; fine-tuning data is judged on how precisely it represents the target task.

Pretraining establishes the model’s general vision-language-action prior across a wide range of tasks and scenes. Fine-tuning takes that pretrained model and specializes it on one target task or embodiment, using a much smaller, precisely scoped batch. See pretraining vs mid-training vs post-training data for robots for the full three-stage breakdown, including where mid-training fits between the two.

A collection built for pretraining is planned for coverage — how many scenes, tasks, objects, and people it spans. A collection built for fine-tuning is planned for precision — how closely it matches the exact deployment task. The two are not interchangeable, and a program spec should say which stage a batch is for.

NVIDIA’s GR00T N1 paper describes training its foundation model on a pyramid of data sources at different scales and levels of realism, from broad web and human video data at the base to real robot trajectory data at the top. That structure is a useful way to think about a VLA pretraining plan: the broadest, most diverse data sits earliest in training, and the narrowest, most task-specific data sits latest.

Egocentric human video sits in the middle of that pyramid — more realistic and physically grounded than web video, but far more scalable to collect than real robot teleoperation data. That is the layer a custom egocentric collection is built to fill.

Egocentric video puts the camera at the embodiment carrying out the task — a head, chest, or glasses-mounted view — so the viewpoint matches what a robot or embodied agent will see at inference time, keeping hands and the instructed object in frame at the moment of contact. Combined with action labels captured live against synchronized depth, it gives a VLA model both the visual prior and the action signal it needs, at a scale that teleoperated robot data alone cannot practically reach.

NVIDIA’s EgoScale paper reports pretraining a robot foundation model on a 20,854-hour corpus of action-labeled egocentric human video, and found a log-linear scaling law between egocentric pretraining data volume and downstream task success — with a 54% success-rate improvement on a real 22-DoF robotic hand after adding a small amount of robot fine-tuning data on top. That result is specifically about egocentric human video used for pretraining, not robot-collected data.

A task instruction paired to an episode is what lets a VLA model learn the mapping from language to action, not just from vision to action. Without an instruction attached at the episode or step level, a video-and-action pair can show what happened, but not what the model was being asked to do — the leg of the triple a VLA model actually conditions on.

That instruction is attached on the same synchronized clock as the video and action stream, so language, vision, and action are provably simultaneous within one episode, the same structural requirement described in training data for VLA models.

Cameras, depth, and IMU are calibrated together across the capture rig, and every stream shares a single hardware clock, so video, action labels, and language annotation are provably aligned rather than approximately synced after the fact. Stereo depth is added where the spec calls for 3D scene geometry, and 3D hand pose is captured with a per-joint visibility flag so occluded fingers are marked rather than guessed.

These are the same requirements used across Firsthand’s egocentric pretraining data and action-labeled egocentric video pages — pretraining data for a VLA model is not a separate capture pipeline, it is the same synchronized, calibrated capture applied at pretraining breadth.

A public egocentric corpus like Egocentric-10K is available immediately under its stated license, but its scenes, tasks, and label formats were set by the dataset’s publisher, not by your program’s spec. A custom collection is scoped to the exact tasks, environments, and label vocabulary your VLA model needs to pretrain on, at a timeline and volume set by a written spec rather than a fixed, published corpus size.

Many programs mix both: a public corpus for early experimentation, then a custom collection scoped to fill the specific gaps — objects, environments, or action vocabularies — the public data does not cover.

Start with the task and scene diversity a pretraining stage needs to cover, the action-label vocabulary (hand pose, contact, segments), and whether language annotation should be written to a fixed instruction set or freeform. That becomes a written spec, quoted for reach, timeline, and price before capture begins.

Many programs start with a Spec Pilot — a small, fast first batch that proves coverage and label quality before scaling volume — then run through the same seven-stage pipeline used for every Firsthand collection, with a median 48-hour first validated batch where coverage is deep, delivered in RLDS, LeRobot, WebDataset, or HDF5 under a buyer-owned perpetual license.