How Are Vision-Language-Action Models Trained?

There is no single recipe. The papers cited on this page describe related but different pipelines, so this guide attributes each stage to the paper that describes it instead of presenting one universal sequence. A vision-language-action (VLA) model takes camera images and a language instruction and outputs an action for a robot to execute.

Read across the sources and a pattern appears. A pretrained vision-language model supplies visual and language knowledge. Robot data teaches the model to act. Some papers add a human-video or mid-training stage. A smaller, task-specific stage then adapts the result to a robot and a job. The table maps each stage to the paper that describes it.

The words pretraining, mid-training and post-training are not used the same way everywhere. GR00T N1 uses pre-training and post-training, EgoScale uses human pretraining followed by mid-training, and OpenVLA and Octo describe pretraining followed by fine-tuning. Compare stages by what data they use, not by their names.

For most of the models here, the starting point is a vision-language model (VLM) already trained on Internet-scale data. The RT-2 paper (https://arxiv.org/abs/2307.15818) studies how such models can be incorporated directly into end-to-end robotic control, and co-fine-tunes them on robotic trajectory data together with Internet-scale vision-language tasks. The OpenVLA paper (https://arxiv.org/abs/2406.09246) says OpenVLA builds on a Llama 2 language model combined with a visual encoder that fuses pretrained features from DINOv2 and SigLIP. Its GitHub README (https://github.com/openvla/openvla) says the openvla-7b checkpoint was trained from the Prismatic prism-dinosiglip-224px VLM.

The π0 paper (https://arxiv.org/abs/2410.24164) proposes a flow matching architecture built on top of a pre-trained VLM to inherit Internet-scale semantic knowledge. The GR00T N1 paper (https://arxiv.org/abs/2503.14734) describes a dual-system model whose vision-language module uses the Eagle-2 VLM, pretrained on Internet-scale data. In post-training it keeps the language component of that backbone frozen and fine-tunes the rest of the model.

Two sources do not fit the pattern. The Octo paper (https://arxiv.org/abs/2405.12213) describes Octo as a generalist robot policy, a transformer that tokenizes language through a pretrained t5-base model and images through a shallow convolution stack. It does not describe Octo as built on a VLM, so this guide does not call it a VLA. The EgoScale abstract (https://arxiv.org/abs/2602.16710) says it trains a VLA model but does not state the backbone in the part read for this page.

For a data team the practical point is that this stage uses web-scale vision-language data you do not collect. Your robot-data decisions start after the backbone is chosen.

Pretraining in these papers means training on a large, varied collection of demonstrations so the model learns a broad mapping from observations to actions. OpenVLA is described as a 7B-parameter open-source VLA trained on a diverse collection of 970k real-world robot demonstrations, and its README says the pretraining mixture comes from Open X-Embodiment datasets in RLDS format. The π0 paper trains on a large and diverse dataset from multiple dexterous robot platforms, including single-arm robots, dual-arm robots and mobile manipulators, and evaluates the model zero-shot after pretraining.

Octo shows how much curation sits inside this stage. The paper trains on 800k trajectories curated from Open X-Embodiment and describes a mixture of 25 datasets. It removed datasets with no image streams or without delta end-effector control, and also removed datasets it judged too repetitive, low resolution or excessively niche. It zero-pads missing camera channels and aligns gripper action spaces across datasets.

GR00T N1 frames pretraining as a data pyramid. Large quantities of web data and human videos form the base, synthetic data from simulation and generated video forms the middle, and real-world robot data forms the top. The paper says the lower layers provide broad visual and behavioral priors while the upper layers ground the model in real-robot execution, and it pretrains end to end by sampling batches across the whole mixture. It also names the problem it is solving: robot data from different embodiments forms an archipelago of data islands, not one coherent corpus.

What this means for data: pretraining rewards breadth, consistent camera and action definitions, and a documented way to reconcile different sources. A pretraining set is judged on coverage first, then on volume.

Some papers describe one; others do not. EgoScale (https://arxiv.org/abs/2602.16710) trains a VLA model on over 20,854 hours of action-labeled egocentric human video, reports a log-linear scaling law between human data scale and validation loss, and introduces a two-stage recipe: large-scale human pretraining followed by lightweight aligned human-robot mid-training. The abstract reports that the final policy improves average success rate by 54% over a no-pretraining baseline using a 22 DoF dexterous robotic hand.

GR00T N1 (https://arxiv.org/abs/2503.14734) uses human video differently. Its human egocentric video comes from open datasets and has no ground-truth actions, so the paper learns a latent-action codebook and uses a trained inverse dynamics model to infer pseudo-actions, which lets it treat action-less video as additional embodiments during training.

For RT-2, OpenVLA, π0 and Octo, the abstracts and README read for this page describe no separate mid-training stage, so this guide states none for them. If your own pipeline adds an intermediate stage, the useful question is what it is aligning: human motion to robot action, one embodiment to another, or one task distribution to another.

Fine-tuning starts from the pretrained weights and updates them on a smaller dataset. The OpenVLA paper shows OpenVLA can be fine-tuned for new settings and fine-tuned on consumer GPUs with low-rank adaptation; the README lists full, partial and LoRA modes, and says a custom dataset can be converted to RLDS to use its fine-tuning script. The Octo paper finetunes the full model on a small target-domain dataset of around 100 trajectories, and says new observations or action spaces can be added through lightweight adapters without changing the pretrained transformer weights. The π0 abstract evaluates the ability to acquire new skills via fine-tuning.

GR00T N1 calls this phase post-training: it fine-tunes the pre-trained model on the data of each single embodiment, with the language component of the backbone frozen. For its real-world benchmarks, the paper says human operators collected task-specific data for durations ranging from 15 minutes to 3 hours depending on task complexity, and low-quality trajectories were filtered out. In simulation it post-trains with 30, 100 and 300 demonstrations per task.

GR00T N1 also records a cost of narrow post-training data. In one qualitative example the pretrained checkpoint used its left hand to grasp an object and hand it to the right hand, while the post-trained checkpoint failed, because all post-training data involved the right hand without any inter-hand transfer. A fine-tuning set narrows behavior, so a good spec names what the model must keep as well as what it must learn.

Action representation is where the listed models differ most. RT-2 expresses actions as text tokens and incorporates them directly into the training set in the same way as natural-language tokens, so one model emits both language and actions. The OpenVLA README describes vanilla OpenVLA-style 256-bin action discretization and says current OpenVLA models are trained to generate actions alone. It also notes that the FAST action tokenizer compresses action chunks into fewer tokens than that discretization.

The other sources use continuous generative outputs. π0 uses a flow matching architecture. GR00T N1 uses a diffusion transformer trained with action flow matching that cross-attends to the VLM output tokens, uses embodiment-specific encoders and decoders for variable state and action dimensions, and processes actions in chunks. Octo uses a diffusion action head that predicts a chunk of several consecutive actions, and its paper reports that this outperformed MSE action heads and discretized action distributions in its own zero-shot and finetuning evaluations. That is a finding about Octo, not a rule for every model.

• VLM backbone: web-scale vision-language data, which a robotics team normally inherits instead of collecting.
• Robot or cross-embodiment pretraining: broad, varied demonstrations with consistent camera and action definitions. Octo reports that only 27% of its pretraining data contains wrist-camera information and only 56% contains language annotations, and links weaker wrist-camera and language-conditioned results to those gaps.
• Human-video or mid-training stage: either action-labeled egocentric video (EgoScale) or action-less video that receives latent or pseudo-action labels (GR00T N1), plus aligned human-robot data where a recipe calls for it.
• Fine-tuning or post-training: a narrow set matched to the deployment task, quality-filtered, and specified so it does not erase behavior you need to keep.
• Action representation: tokens, diffusion and flow-matching heads all consume action labels, so the label definition and the mapping to your robot must be fixed before collection.

The table below maps those needs to what this site states Firsthand delivers. It lists only fields the site states and marks the rest Not stated.

Two of the listed papers use it. EgoScale reports that large-scale action-labeled egocentric human video can serve as a supervision source for a VLA, and that its validation loss correlates with downstream real-robot performance. GR00T N1 places human egocentric video at the base of its data pyramid and gets actions for it from latent-action and inverse-dynamics labels. Both papers report results on their own data, not on Firsthand data, and nothing here suggests otherwise.

The caution is the one GR00T N1 itself implies: human video has no robot actions, so someone has to decide how hand motion becomes an action your model can learn. That is why the Firsthand site treats the mapping as a specification decision made before capture. Whether human-side data helps your model is something to test against your own robot and tasks.

Firsthand collects custom egocentric data on demand, to a written spec, for teams that need data a public mixture does not cover. The only language field the site ties to Firsthand capture is one task instruction per episode, kept on the same clock as the video and action stream. Each episode also carries 3D hand pose, contact events and action segments, and data is delivered in RLDS, LeRobot, WebDataset or HDF5 under a perpetual, buyer-owned license.

• Name the stage first. Breadth-oriented data for pretraining and a narrow task set for fine-tuning are specified differently.
• Write the spec: the tasks, objects and accept criteria, plus the behaviors a fine-tuned model must keep.
• Settle the mapping from human hand signals to your robot action space before capture, not after delivery.
• Pick the delivery format your training code already reads.
• Request a sample, check it against your pipeline, then scale the collection.

Teams that want to move fast on data do it by settling the spec early, not by skipping it. The pretraining and fine-tuning collection pages linked below describe each case, and the custom dataset request starts the written spec.