Glossary
What Is a Vision-Language-Action Model?
Last reviewed: September 2026
Short answer
A vision-language-action (VLA) model takes in a visual scene and a language instruction and outputs an action — a robot policy, gripper command, or hand trajectory. That mapping only exists in training data if each episode pairs the visual frames with an instruction and the action stream that carried it out, all on a timeline where the three are provably simultaneous.
01
Why it matters
A video-caption pair with no action stream, or an action log with no instruction, is missing one leg of the triple a VLA model needs — it can only learn the mapping from vision and language to action if all three are present and synchronized in the same episode.
02
How Firsthand handles it
Firsthand captures synchronized egocentric video, a paired task instruction, and the action that carried it out — 3D hand pose, gripper state, or a robot action log — hardware-triggered onto one clock, to a written spec, on demand.
03
What a VLA model trains on, mapped to the streams and fields that provide it
| Input to the model | What it needs from the dataset | Where that lives in a Firsthand episode |
|---|---|---|
| Vision | Dense egocentric RGB, and stereo depth where the spec calls for it | Head, wrist, or chest-mounted camera streams |
| Language | A task instruction paired to the episode | Instruction metadata attached at the episode or step level |
| Action | 3D hand pose, gripper/contact state, or robot action logs on the same clock as the video | action_segment fields, hardware-synchronized to the video streams |
05
Frequently asked questions
- What is a vision-language-action (VLA) model?
- A VLA model takes a visual scene and a language instruction as input and outputs an action — a robot policy, gripper command, or hand trajectory. It learns the mapping from vision and language to action from training episodes where all three are present together.
- What does a dataset need to provide to train a VLA model?
- Each episode needs egocentric video, a task instruction, and the action stream that carried it out — hand pose, gripper state, or a robot action log — all captured on one synchronized clock. See training data for VLA models for the full breakdown.
- Is a VLA model the same as behavioral cloning?
- No. Behavioral cloning is a training method that maps observation to action; a VLA model is an architecture that also conditions on a language instruction. A VLA model can be trained with a behavioral-cloning objective, but the two terms describe different things — one an architecture, one a training method.
- Why does egocentric video matter for VLA training?
- Egocentric video puts the camera at the embodiment carrying out the instruction, so the viewpoint matches what a robot or embodied agent will see at inference time, keeping hands, tools, and the instructed object in frame at the moment of contact. See training data for VLA models for more.
- What format should VLA training data be delivered in?
- Any episodic, step-based format that preserves the alignment between video, instruction, and action works — RLDS and LeRobot format are both built around that structure, and WebDataset works for teams streaming sharded tars at scale.