VLA vs Diffusion Policy: What Is the Difference?

A vision-language-action (VLA) model is defined by what it is built from and what it is conditioned on. The RT-2 abstract (https://arxiv.org/abs/2307.15818) proposes incorporating vision-language models trained on Internet-scale data directly into end-to-end robotic control, and names the resulting category vision-language-action models. A VLA takes a visual observation and a language instruction and outputs an action, with a pretrained vision-language component supplying the language side.

A diffusion policy is defined differently: by how it represents the action, not by what it is conditioned on. The Diffusion Policy paper (https://arxiv.org/abs/2303.04137) represents a robot’s visuomotor policy as a conditional denoising diffusion process on the action space, conditioned on visual observations. The paper’s own policy is not language-conditioned; its project page (https://diffusion-policy.cs.columbia.edu/) and GitHub repository (https://github.com/real-stanford/diffusion_policy) describe benchmark and real-world tasks driven by visual observations, not a language instruction.

These are two different axes, not two competing camps. A model can be a VLA and also use a diffusion-style action output, and the GR00T N1 paper (https://arxiv.org/abs/2503.14734) is exactly that case: a vision-language backbone paired with a Diffusion Transformer module that generates actions through flow-matching. This page compares the two concepts along both axes — conditioning and action representation — and does not call either one the winner.

This page does not claim that Firsthand data trained, fine-tuned or evaluated any named model, and it claims no affiliation with the authors of the models it cites. Third-party statements come only from the official papers linked in Sources. Everything about Firsthand comes from other pages of this site, and where the site states nothing, this page says Not stated.

A VLA takes a visual observation and a language instruction as input and outputs an action for robot control. RT-2 builds this by co-fine-tuning vision-language models on robotic trajectory data together with Internet-scale vision-language tasks, and expresses actions as text tokens incorporated into the same training set as natural language tokens. The π0 paper (https://arxiv.org/abs/2410.24164) describes a flow matching architecture built on top of a pre-trained vision-language model (VLM) to inherit Internet-scale semantic knowledge, with a separate action expert that produces continuous actions via flow matching.

The GR00T N1 paper describes a dual-system design: a vision-language module (its paper calls this System 2) that interprets the environment through vision and language instructions, built on the NVIDIA Eagle-2 VLM, paired with a Diffusion Transformer module (System 1) that generates motor actions. OpenVLA (https://arxiv.org/abs/2406.09246) builds on a Llama 2 language model combined with a visual encoder that fuses pretrained DINOv2 and SigLIP features, and its abstract calls it a 7B-parameter open-source VLA trained on 970k real-world robot demonstrations.

What these four sources share is a pretrained vision-language starting point and a language instruction as an input the model is conditioned on. That conditioning on language, through a vision-language backbone, is what the sources use to define the category, not any one way of producing the output action.

The Diffusion Policy paper represents a robot’s visuomotor policy as a conditional denoising diffusion process on robot action space. Instead of directly outputting an action, the policy infers the action-score gradient, conditioned on visual observations, over a series of denoising iterations. The paper reports this formulation handling multimodal action distributions, scaling to high-dimensional action spaces, and training stably, and benchmarks it across 15 tasks from 4 robot manipulation benchmarks.

The paper’s policy is conditioned on visual observations and, where used, a proprioceptive state — not on a language instruction. The paper also notes that goal conditioning is possible with the same FiLM conditioning method used for observations, but this is goal conditioning, not language conditioning, and the paper does not describe a language-instruction input or a vision-language backbone.

Diffusion decoding also appears inside VLA-adjacent models for a different purpose: representing the action, not conditioning on language. The Octo paper (https://arxiv.org/abs/2405.12213) describes a transformer-based policy that tokenizes task descriptions with a pretrained language model and observations with a lightweight CNN, and produces actions with a diffusion head; its own text calls Octo a generalist robot policy, not a VLA, and this page keeps that distinction. The GR00T N1 Diffusion Transformer module is a third variant again: a VLA’s action output, built with flow-matching rather than the original DDPM formulation.

Separating the two axes clarifies the comparison. Conditioning is what decides the behavior: a VLA is conditioned on a language instruction, via a pretrained vision-language component, in addition to the visual observation. The original Diffusion Policy is conditioned on the visual observation (and optionally proprioceptive state), with no language instruction described in the sources read.

Action representation is how the output is produced: the original Diffusion Policy denoises a sequence of continuous actions from Gaussian noise using DDPM. RT-2 instead expresses actions as discrete text tokens. OpenVLA follows the same discretization approach as RT-2, mapping each action dimension into one of 256 bins by quantile and predicting them as tokens in the language model’s vocabulary — a different representation from the original Diffusion Policy’s continuous denoising. π0 and GR00T N1 use flow matching, a variant of diffusion that the π0 paper distinguishes from autoregressive discretization, each with their own action expert or Diffusion Transformer module. The Octo paper uses the original DDPM-style diffusion head described in the Diffusion Policy paper, cited directly in Octo’s own text.

The two tables below separate these axes. The first sets the generic categories side by side on inputs, conditioning, action representation and typical data. The second checks each named source against the same four questions, and marks a cell Not stated where the source read for this page does not say.

Not in the paper that defines the term. The Diffusion Policy paper’s formulation conditions the denoising process on visual observations, and the paper describes goal-image conditioning as an alternative, not a language instruction. Its project page and GitHub repository describe the benchmarked tasks (Push-T, mug flipping, sauce pouring and spreading, and the bimanual real-world tasks) by visual goal, not by a natural-language command.

Separately, several models combine language conditioning with a diffusion-style action head, which is where the two axes meet. The Octo paper describes a model that can be instructed via language commands or goal images, with a diffusion head used only to decode the action, not to decide what the robot should do. The GR00T N1 paper’s language instruction goes into the Eagle-2 vision-language backbone, and the Diffusion Transformer module that follows is conditioned on that backbone’s output tokens together with the robot’s state — so the language conditioning and the diffusion-style action output are two separate components of the same model, not one mechanism.

Reading only the Diffusion Policy paper, a diffusion policy and language conditioning are independent choices: the original paper does not use one, and later VLA-category models that do condition on language add it through a vision-language backbone, with diffusion or flow matching handling only the action output.

The data differs with what the model is conditioned on, not with how it decodes actions. A VLA example is a sequence: a visual observation, a language instruction, and the action that carried the instruction out, on a timeline where all three are provably simultaneous. RT-2 trains on robotic trajectory data together with Internet-scale vision-language tasks. OpenVLA’s abstract names 970k real-world robot demonstrations. π0’s paper describes pre-training on a large and diverse dataset from multiple dexterous robot platforms — single-arm, dual-arm and mobile manipulators — combining task names and segment annotations as language labels, with 903 million timesteps from its own datasets at 68 tasks plus a share of the Open X-Embodiment dataset. GR00T N1 trains on a heterogeneous mixture of real-robot trajectories, human videos, and synthetically generated datasets.

The original Diffusion Policy paper’s own benchmarks need only visual observations (and sometimes proprioceptive state) paired with the demonstrated action sequence — no language instruction is part of that training example, because the policy is not conditioned on one. Octo, which pairs language-or-goal-image conditioning with a diffusion action head, trains on 800k trajectories curated from the Open X-Embodiment dataset, and its paper reports that only 56% of its pretraining data contains language annotations, with the rest trained via goal-image conditioning instead.

None of these volumes is a guide for sizing a custom collection: each describes that paper’s own dataset, under its own definition of a task and a trajectory, and this page states no episode count or hour count for a Firsthand custom collection, because the site gives none for one. A second difference, independent of volume, is synchronization: whichever path a team takes, the instruction (if used), the vision, and the action all have to share one clock, or the model learns the wrong mapping.

This page does not pick a winner, and the sources read here do not support one: each paper reports its own tasks, its own robots, and its own evaluation protocol, so a result in one paper is not a result in another. What the sources do make clear is that the choice is really two separate questions. First, does the task need a language instruction, or is a fixed task, a goal image, or a single-purpose deployment enough? If language matters, that points toward a VLA-category model with a vision-language backbone. Second, does the action distribution need to be multimodal or high-dimensional? The Diffusion Policy paper reports that its denoising formulation handles multimodal action distributions and high-dimensional action spaces, which is why diffusion or flow-matching action heads show up inside several of the VLA-category models in this comparison, not as a substitute for the language question but as an answer to a different one.

A team is not choosing one category and discarding the other: GR00T N1 and π0 show that language conditioning and a diffusion-style or flow-matching action output can be combined in the same model. The practical decision is which pretrained components and which training recipe fit the task, the available compute, and the data a team can collect or source — not which name sounds newer.

Write down which conditioning your model needs — a language instruction, a goal image, or neither — and whether the action output will be tokenized, denoised, or flow-matched, since that shapes what the training example has to contain. Then name the tasks and objects, the setting, the wording of the single task instruction for each episode if one is used, which human signals your pipeline will use, the delivery format, and how many episodes to hold back for evaluation. We turn that into a written spec covering tasks, environments, capture tier, annotation depth and accept criteria, then quote reach, timeline and price before any capture begins.

The site publishes two timing figures. Where coverage is already deep, the first validated batch lands in a median of 48 hours from a signed spec. Where coverage is thin, it typically takes three to four weeks, because vetting operators there is the bottleneck, and the quote says which applies. Neither is a delivery date for your dataset. The usual route starts with a Spec Pilot, a small, fast batch that proves the spec and accept criteria on real footage before you scale.

Delivery formats are RLDS, LeRobot and WebDataset, plus HDF5, zarr and Rerun .rrd, each a projection of the same episode schema. Every episode links to a signed contributor release through a documented consent chain, and faces, plates and screens are detected and irreversibly blurred at ingest. Billing is per validated hour, a reject log ships with every batch, and rejected episodes are recaptured at our cost. Every collection ships under a perpetual, buyer-owned commercial license.