Glossary
What Is CoT-VLA?
Last reviewed: October 2026
Short answer
CoT-VLA is a 7B vision-language-action model from arXiv 2503.22020 that adds visual chain-of-thought reasoning: it first predicts a future subgoal image, then generates a short action sequence to reach it. It is built on VILA-U and also learns from action-less video. It is third-party research, not Firsthand’s.
01
What is CoT-VLA?
CoT-VLA is introduced in arXiv 2503.22020 by Qingqing Zhao and co-authors. The abstract says current VLAs map inputs directly to outputs and lack intermediate reasoning, and proposes predicting future image frames as visual goals before a short action sequence. Read: abstract, full text and project page; no official GitHub repository was identified.
02
Where does Firsthand fit next to CoT-VLA?
CoT-VLA is third-party research. Firsthand is not affiliated with its authors and claims no use, training or compatibility with it. Firsthand captures first-person human video, not robot data, fast, on demand and to a written spec: 21 keypoints per hand with a visibility flag, wrist trajectories and action segments, plus one task instruction per episode.
03
What is visual chain-of-thought and how are subgoal images used?
In language models, chain-of-thought means writing intermediate steps before an answer. The paper applies that to robot control: given the current observation and instruction, the model first generates an image of the planned state n frames ahead, then a short action sequence conditioned on it, closed-loop. The authors say subgoal images already exist in demonstration data and need no action annotations, so the reasoning step can also learn from action-less video.
04
How is CoT-VLA trained, as stated in the sources?
Pretraining mixes robot demonstrations with action-less video. The subgoal step trains on both; the action step trains on robot demonstrations only. The vision tower stays fixed while the LLM backbone, projector and depth transformer train. Adaptation then fine-tunes on demonstrations collected on the target robot.
05
Which robots and tasks does the paper report, as stated in the sources?
Three settings: the LIBERO simulation benchmark, Bridge-V2 on a 6-DoF WidowX arm, and Franka-Tabletop on a Franka Emika Panda 7-DoF arm. The abstract reports gains of 17% in real-world manipulation and 6% in simulation benchmarks over the state-of-the-art VLA. These are the authors’ own figures.
06
How does CoT-VLA relate to embodied chain-of-thought and other VLA models?
The paper says earlier intermediate steps, such as language descriptions, keypoints or bounding boxes, often need extra pre-processing pipelines, and cites embodied chain-of-thought among text-plan and bounding-box methods. Its baselines are Diffusion Policy, OpenVLA, Octo and SUSIE. It names limits: 256 image tokens before action tokens, and an average 7x slowdown at chunk size 10.
07
What training data do VLA models need, and how do you get custom egocentric data for your own tasks?
CoT-VLA’s action step needs robot demonstrations with actions, which Firsthand does not deliver; egocentric human video as CoT-VLA training data is Not stated in the sources read. Firsthand captures human first-person video. Put tasks and signals in a written spec and confirm it on a Spec Pilot. The median from signed spec to first validated batch is 48 hours where coverage is deep and three to four weeks where thin. Formats: RLDS, LeRobot v2.1, WebDataset, HDF5; mapping to CoT-VLA’s input is the buyer’s pipeline.
08
CoT-VLA architecture, reasoning steps and training data, as stated in the official sources
| Item | What the official sources state |
|---|---|
| Model | 7B VLA that understands and generates visual and action tokens (abstract) |
| Base model | VILA-U; images at 256x256 resolution |
| Reasoning step | Subgoal image n frames ahead, generated with causal attention |
| Action step | Short action sequence with full attention; chunk size 10 |
| Robot data | Subset of Open X-Embodiment: third-person views, single-arm end-effector control |
| Action-less video | EPIC-KITCHENS and Something-Something V2 |
10
CoT-VLA beside a generic VLA and embodied chain-of-thought, as each source states
| Item | CoT-VLA (sources read) | Vision-language-action model | Embodied chain-of-thought |
|---|---|---|---|
| Core idea | Subgoal image first, then actions | Vision-language model adapted for robot control | Text reasoning chain before each action |
| Intermediate reasoning | A generated subgoal image | None in the vanilla form, per the CoT-VLA paper | Plan, sub-task, motion, gripper position, object boxes |
| Action-less video | Used for the subgoal step | Not described | Not described |
| Egocentric human video | Not stated in the sources read | Not described | Not described |
11
What CoT-VLA consumes beside what the Firsthand site states
| Aspect | CoT-VLA (sources read) | What the Firsthand site states |
|---|---|---|
| Images | Third-person views for robot data, 256x256 | Head-mounted egocentric video plus two wrist cameras |
| Language | A language instruction or description per sequence | One task instruction per episode |
| Robot actions | Required for the action step | Not delivered; Firsthand captures human video, not robot data |
| Hand data | Not applicable; the embodiment is a robot arm | 21 keypoints per hand, visibility flag, wrist trajectories, action segments |
| File layout | Not stated in the sources read | RLDS, LeRobot v2.1, WebDataset, HDF5 |
| Compatibility | Not claimed | Not claimed |
12
Frequently asked questions
- What is CoT-VLA?
- A 7B VLA from arXiv 2503.22020, built on VILA-U, that predicts a subgoal image before it generates actions.
- What is visual chain-of-thought in CoT-VLA?
- Generating a future image as an intermediate reasoning step, then a short action sequence conditioned on that image.
- How is CoT-VLA trained?
- Pretraining on robot demonstrations and action-less video, then fine-tuning on demonstrations from the target robot.
- Is CoT-VLA open source?
- Not stated in the sources read. The paper and project page text give no code repository or licence.
- Was CoT-VLA trained on egocentric human video?
- Not stated in the sources read. The paper uses robot demonstrations plus EPIC-KITCHENS and Something-Something V2 as action-less video.
- Does Firsthand use CoT-VLA, and can I request a custom dataset?
- Firsthand is not affiliated with the CoT-VLA authors and claims no use, training or compatibility. You can request a custom dataset, confirmed first on a Spec Pilot.
Related terms
Sources