VLA vs VLM: What Is the Difference?

A vision-language model (VLM) takes images and text together and produces language. A vision-language-action (VLA) model takes a visual observation and a language instruction and produces an action for robot control. That added action output is the whole difference in one line, and the rest of this page follows it through what the model is built from, what it trains on and what data a team has to collect.

The two are related by construction, not set against each other. The RT-2 abstract (https://arxiv.org/abs/2307.15818) studies how vision-language models trained on Internet-scale data can be incorporated directly into end-to-end robotic control, and names the resulting category vision-language-action models. The π0 abstract (https://arxiv.org/abs/2410.24164) describes a flow matching architecture built on top of a pre-trained vision-language model. In the sources read for this page, a VLA is a vision-language starting point that has been extended and trained to act.

Where the two diverge is the data. A VLM learns from images paired with language that is true about them. A VLA also needs the action that carried the task out, on a timeline where the vision, the instruction and the action are provably simultaneous. That is what makes VLA training data harder to source than VLM data, and it is where Firsthand’s custom egocentric collection fits.

This page does not claim that Firsthand data trained, fine-tuned or evaluated any named model, and it claims no affiliation with the authors of the models it cites. Third-party statements come only from the official papers linked in Sources. Everything about Firsthand comes from other pages of this site, and where the site states nothing, this page says Not stated.

A vision-language model combines an image encoder with a language model, so it can interpret a scene and a written request together. Images and text go in. What comes out is language: an answer, a description or a decision expressed in words. A VLM does not output an action by itself, which is why a robot cannot be driven directly from one.

VLM training data is images or video frames paired with language that is true about them. The RT-2 abstract names visual question answering as one example of the Internet-scale vision-language tasks such models are trained on. The pairing is what the model learns, so footage with no true language, or language that describes something other than the footage, teaches the wrong relationship. The language side is where VLM datasets differ most.

This page describes VLMs generically. It does not name or compare VLM products. The only VLM it names is one that a listed paper itself names: the GR00T N1 paper (https://arxiv.org/abs/2503.14734) states that it uses the NVIDIA Eagle-2 VLM as the vision-language backbone.

On the Firsthand side, the one language field the site states is a task instruction paired to each episode. Captions, narration, image-text pairs, visual question answering and dialogue are not stated as deliverables, so none is claimed. That single instruction is the whole of the language data this page describes.

A VLA adds an action output. Instead of ending in text, the model ends in something a robot can execute. The RT-2 abstract describes one way to do it: the actions are expressed as text tokens and incorporated directly into the training set of the model in the same way as natural language tokens. That keeps one token format for language and for action.

Other sources attach the action output differently. The π0 abstract describes a flow matching architecture. The GR00T N1 paper describes a Diffusion Transformer module, trained with action flow-matching, that generates motor actions from the vision-language module’s output tokens together with the robot’s state. The Octo paper (https://arxiv.org/abs/2405.12213) describes a transformer-based policy with a diffusion head to model action distributions. The Octo paper calls Octo a generalist robot policy, and this page uses that wording because the text read here does not call it a VLA.

The inputs grow as well. In the GR00T N1 paper the model takes the robot’s state, visual observations and a language instruction as input, and its output is the corresponding motor action. A VLA’s input can therefore include robot state in addition to image and text, which a VLM has no use for.

Not every source states the same things. The OpenVLA abstract (https://arxiv.org/abs/2406.09246) and the EgoScale abstract (https://arxiv.org/abs/2602.16710) do not describe how the action is encoded, so the tables below say Not stated for those cells rather than filling the gap.

Four of the listed sources describe a vision-language or language-model starting point, and each says it differently. RT-2 co-fine-tunes state-of-the-art vision-language models on both robotic trajectory data and Internet-scale vision-language tasks. π0 builds a flow matching architecture on top of a pre-trained VLM so that it inherits Internet-scale semantic knowledge. GR00T N1 contains a vision-language backbone and a DiT-based flow-matching policy, and names Eagle-2 as the backbone. OpenVLA builds on a Llama 2 language model combined with a visual encoder that fuses pretrained features from DINOv2 and SigLIP, and its abstract calls it a 7B-parameter open-source VLA trained on a diverse collection of 970k real-world robot demonstrations.

Octo and EgoScale are described separately. The Octo paper tokenizes task descriptions with a pretrained language model and observations with a lightweight CNN, passes them through a transformer backbone, and produces actions with diffusion decoding. It reports training on 800k trajectories curated from the Open X-Embodiment dataset. This page treats Octo as the paper does, a generalist robot policy, and does not call it VLM-based. The EgoScale abstract says it trains a VLA on over 20,854 hours of action labeled egocentric human video, and does not state the backbone.

The pattern across these sources is a split of labour. The pretrained vision-language part supplies what the abstracts call Internet-scale knowledge, and robot or action data supplies the mapping to motion. The vision-language part is available pretrained. The action data is the part a team usually has to source for its own robot and task, which is why the rest of this page is about it.

The second table below sets each source against the same four questions: how it relates to a VLM, how it outputs actions, what data it names, and where the claim comes from. Cells that the source does not state say Not stated.

The two differ in what a training example is. A VLM example is an image, or a few frames, plus language. A VLA example is a sequence: an observation, an instruction and an action, in time order. The first table below sets the two side by side across inputs, outputs, typical data and target.

The sources also show that VLA data is rarely action data alone. RT-2 trains on robotic trajectory data together with Internet-scale vision-language tasks. GR00T N1 trains on a heterogeneous mixture of real-robot trajectories, human videos and synthetic datasets. In the sources read here, action data is added to a vision-language base, not substituted for it.

Volumes in the sources describe each paper’s own dataset, and none is a guide for a custom collection. The OpenVLA abstract names 970k real-world robot demonstrations. The Octo paper names 800k trajectories from Open X-Embodiment. The EgoScale abstract names over 20,854 hours of action labeled egocentric human video. This page states no episode count or hour count for a Firsthand custom collection, because the site gives none for one. The size of a collection is set in the written spec and the quote.

A second difference is synchronization. A caption can be attached to a frame loosely and still teach the model something. An action cannot: a hand or gripper movement that is a few frames out of step with the frame it belongs to teaches the wrong mapping. VLA data therefore depends on all streams sharing one clock, which is a collection property and not something a team can add afterwards.

A VLA needs four things beyond what a VLM needs. First, an action stream: a record of what the actor did, in a form the model can learn to output. Second, one clock across vision, instruction and action, so the three are provably simultaneous. Third, an instruction tied to the episode, so the model learns which behaviour answers which request. Fourth, a decision about how the recorded action maps to the actions your robot can execute.

The fourth point matters most for egocentric human data. The action in a Firsthand episode is human action: 3D hand pose with per-joint visibility, contact events on grasp and release, and frame-accurate action segments. The official sources describe models that output robot actions, and the GR00T N1 paper describes embodiment-specific encoders and decoders to handle variable state and action dimensions. The site does not state that Firsthand delivers robot action logs, teleoperation data, robot embodiments or action spaces, so those are Not stated. If your stack needs robot actions, settle in the spec how the human signals are to be mapped, and who does the mapping, before you plan around the data.

The third table below maps each need to what the site states Firsthand delivers and where it lives in a delivery. The site states ten streams synced under 2 ms with head and wrist RGB, depth, 3D hand pose, contact events and action segments, plus 200 Hz 6-DoF IMUs on the Firsthand capture rig. Whether a given collection includes the IMU streams is set in the spec.

Coverage is also different. A VLM is judged on breadth of language and imagery. A VLA collected for a deployment is judged on how closely the data matches the task and setting. The site states that a spec can name the failure modes and recovery behaviours a policy currently struggles with, so a batch is built around those and not only the clean version of the task.

Two of the listed sources train on human video. The EgoScale abstract says a VLA was trained on over 20,854 hours of action labeled egocentric human video, and the GR00T N1 paper lists human videos in its heterogeneous training mixture alongside real-robot trajectories and synthetic datasets. Both are statements about those papers’ own data and models.

What the sources do not say is that any listed model depends on data of the kind Firsthand collects. This page does not claim that Firsthand data was used by, or would reproduce the results of, EgoScale, GR00T N1 or any other model. It does not claim any affiliation with their authors.

What the site does state is what Firsthand captures. Egocentric capture is head-mounted or wrist-mounted and matched to the target task, delivered as episodes with synchronized video and depth, 3D hand pose, contact events, action segments and one task instruction. That is the same family of signal that the EgoScale abstract describes as action labeled egocentric human video, but the site publishes no label vocabulary, so a team should confirm the exact labels in the spec.

Human video and robot actions are different things. A team that wants to train a VLA on egocentric human data decides, ahead of collection, whether its pipeline uses hand pose directly, retargets it to a robot, or uses the data in another way. That decision belongs in the spec, since it determines which signals are worth capturing.

Write down the model and the stage it is at: a VLM-based starting point that needs action data, a pretrained VLA that needs fine-tuning data, or an earlier pretraining effort. Then name the tasks and objects, the setting, the wording of the single task instruction for each episode, which human signals your pipeline will use, the delivery format and how many episodes to hold back for evaluation. We turn that into a written spec covering tasks, environments, capture tier, annotation depth and accept criteria, then quote reach, timeline and price before any capture begins.

The site publishes two timing figures. Where coverage is already deep, the first validated batch lands in a median of 48 hours from a signed spec. Where coverage is thin, it typically takes three to four weeks, because vetting operators there is the bottleneck, and the quote says which applies. Neither is a delivery date for your dataset. The usual route starts with a Spec Pilot, a small, fast batch that proves the spec and accept criteria on real footage before you scale.

Delivery formats are RLDS, LeRobot and WebDataset, plus HDF5, zarr and Rerun .rrd, each a projection of the same episode schema. The OpenVLA repository (https://github.com/openvla/openvla) states that it supports datasets in RLDS format. Whether Firsthand RLDS field names match what that repository expects is Not stated and untested, so expect a short mapping step.

Every episode links to a signed contributor release through a documented consent chain, and faces, plates and screens are detected and irreversibly blurred at ingest. Billing is per validated hour, a reject log ships with every batch, and rejected episodes are recaptured at our cost. Every collection ships under a perpetual, buyer-owned commercial license.