What Is ChatVLA?

ChatVLA is introduced in ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Model (arXiv 2502.14420), first submitted on 20 February 2025 and last revised on 21 February 2025 by Zhongyi Zhou, Yichen Zhu and colleagues. The paper lists Midea Group, East China Normal University, Shanghai University, the Beijing Innovation Center of Humanoid Robotics and Tsinghua University. The official project page says it was accepted to EMNLP 2025 as a Main Oral and points to a follow-up, ChatVLA-2, which this page does not cover. The abstract opens by asking why large language models cannot replicate the unified way humans perceive, comprehend and act in the physical world. The paper calls ChatVLA the first work to unify multimodal understanding and embodied control in a single end-to-end network. This page draws on the paper, the official project page and the GitHub repository the project page links. No Hugging Face paper page and no Hugging Face model card were read.

The paper compares three ways to train a VLA: robot demonstration data only, robot data with reasoning phrases added, and co-training with visual-text pairs. Using DiffusionVLA as the test model, it reports that robot data alone erodes conversational ability entirely, with a score of 0 on all six understanding benchmarks. Reasoning data partly preserves understanding, and visual-text pairs weaken control. From this the authors name two challenges. Spurious forgetting is where the alignment between actions and visual-text data is fragile and gets overwritten during fine-tuning, so the loss may be a shift in alignment rather than lost knowledge. Task interference is where control and understanding tasks have conflicting parameter spaces and degrade each other when trained together. The analysis sampled 54k image-text pairs from LLaVA, followed RT-2 with a 3:1 ratio of robot to visual-text data, and reports 112 trials across five real-world robot tasks.

The paper names Qwen2-VL-2B as the vision-language backbone and says the action head follows DiVLA. The README says the Qwen2-VL weights are used off the shelf, with no post-training on the VLM, and that the official config file is replaced with one designed for VLA. The Mixture-of-Experts sits on the MLP layers. A control expert and an understanding expert process robot data and visual-text data separately, while shared self-attention layers let knowledge transfer between them. The paper describes a dual router and distinct system prompts, "Answer based on question" for understanding and "Predict robot action" for control, and says only one path is active at inference. The authors cite Dual Coding Theory as the motivation. The exact number of expert modules per layer and inference speed are Not stated in the sources read.

Phased Alignment Training has two stages, inspired by curriculum learning. Stage 1 trains on robot data, with reasoning data included to keep the visual and text parts aligned, and only the control expert and its action head are active. Stage 2 co-trains visual-text and robot data. Both experts use a learning rate of 2e-5. The visual-text data is the LLaVA-1.5 dataset, mixed at a 1:3 ratio of visual-text to robot data. In an ablation, 1:1 and 3:1 ratios scored lower on MMMU and MMStar, and the paper concludes that a small amount of visual-text data is enough to reactivate alignment. Batch size, step counts, training hardware, robot demonstration counts and training hours are Not stated in the sources read.

Understanding was tested with VLMEvalKit on six multimodal understanding benchmarks and seven VQA benchmarks. In Table 1, the 2B ChatVLA scores 37.4 on MMMU, 47.2 on MMStar, 71.2 on TextVQA and 83.3 on DocVQA. OpenVLA scores 0 on all of them and ECoT scores 5.4 on MMMU. The paper says its backbone uses 3.5x fewer parameters than ECoT. The base Qwen2-VL scores 41.1 and 48.0 on MMMU and MMStar. One sentence in the paper text swaps the MMStar and MMMU figures, so this page follows Table 1 and the abstract.

Control was tested on 25 real-world manipulation tasks with 528 trials, in three groups: long-horizon tasks with direct prompting, long-horizon tasks with a high-level planner, and cross-skill multi-tasking in bathroom, kitchen and tabletop scenes. In Table 4, ChatVLA succeeds in 55 of 107 trials, against 20 of 107 for OpenVLA and 18 of 107 for Octo. The authors attribute the remaining gap to Qwen2-VL on some MMMU categories to expert knowledge missing from LLaVA data.

The project page links the GitHub repository tutujingyugang1/ChatVLA_public, which lists an MIT License. The README covers install, downloading Qwen2-VL weights, data preparation, two training scripts, evaluation and a ChatVLA weights download on Hugging Face, though no model card was read. Robot data is converted to h5py format following DexVLA, and vision-language data uses the LLaVA conversation format. Evaluation builds on VLMEvalKit. Hardware requirements and licence terms for the paper, datasets and weights beyond the repository label are Not stated in the sources read. This is not legal advice.

ChatVLA is an architecture and a training recipe. Its control data is robot demonstrations of state-action pairs, with images, text instructions and robot actions, plus reasoning data and LLaVA visual-text pairs. Firsthand supplies something different: real human egocentric demonstrations from a head-mounted camera and two wrist cameras. Nothing on this site describes robot action or joint data, end-effector pose labels, teleoperation data, simulation data, visual question answering data, instruction-tuning conversations or video captions, and none is claimed here. The only language field the site states is one task instruction per episode.

Firsthand makes no claim that its data trained, was used with or is compatible with ChatVLA, Qwen2-VL, DexVLA, LLaVA or any other model, robot, simulator or benchmark. Firsthand data is real capture, never synthetic, generated or simulated. The site’s general view is in its guide on training data for VLA models, linked below.

Firsthand collects real egocentric demonstrations on demand, to a written spec. The rig is a head-mounted global-shutter camera plus two wrist cameras, with active-stereo depth and 3x IMU on one hardware-triggered PTP clock. Episodes carry 3D hand pose at 21 joints per hand with a per-joint visibility flag, wrist trajectories, contact events, frame-accurate action segments and one task instruction per episode.

You write the spec, confirm it on a Spec Pilot, and pay per validated hour. The site states a 48-hour median turnaround where coverage is deep and three to four weeks where it is thin. Delivery is RLDS, LeRobot, WebDataset or HDF5, with a buyer-owned commercial license. Request a custom dataset below.

What is ChatVLA?

A vision-language-action framework that the paper says unifies multimodal understanding and robot control in one network. It combines Phased Alignment Training with a Mixture-of-Experts design and is introduced in arXiv 2502.14420.

What two problems does ChatVLA address?

Spurious forgetting, where robot training overwrites visual-text alignment, and task interference, where control and understanding tasks degrade each other when trained jointly. The paper reports both from its analysis of three VLA training settings.

How does the mixture-of-experts design work?

Per the paper, a control expert and an understanding expert replace the MLP layers, while self-attention layers are shared. Distinct system prompts mark each task, and only one path is active at inference. The exact number of experts per layer is Not stated in the sources read.

How is ChatVLA trained?

In two stages. Stage 1 trains on robot data with reasoning data and only the control expert active. Stage 2 co-trains visual-text and robot data at a 1:3 ratio. Both experts use a learning rate of 2e-5, per the paper.

Can ChatVLA be used commercially?

The GitHub repository lists an MIT License. Licence terms for the paper, datasets and weights beyond that label are Not stated in the sources read. Check the repository directly; this is not legal advice.

Does Firsthand use ChatVLA, and can I request a custom dataset?

Firsthand is not affiliated with ChatVLA’s authors and makes no claim of use, training or compatibility. Firsthand data is real human egocentric capture. You can request a custom egocentric dataset: write a spec, confirm it on a Spec Pilot, pay per validated hour.