What Is VLAS?

VLAS is introduced in the paper VLAS: Vision-Language-Action Model With Speech Instructions For Customized Robot Manipulation (arXiv 2502.13508) by Wei Zhao, Pengxiang Ding, Min Zhang, Zhefei Gong, Shuanghao Bai, Han Zhao and Donglin Wang, submitted on 19 February 2025 and accepted at ICLR 2025. The authors argue that existing vision-language-action models rely on vision-language models that support text-based instructions only, leaving the more natural speech modality unused. The official GitHub repository is linked from the paper itself.

VLAS is third-party research. Firsthand is not part of it or affiliated with its authors, and makes no claim that any model was trained on Firsthand data. What the site states is custom egocentric collection of real human demonstrations, fast and on demand, to a written spec: 3D hand pose at 21 joints per hand, calibrated stereo depth and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset with a buyer-owned commercial license. The only language data the site states is one task instruction per episode, and the site makes no speech or audio-instruction capture claim.

The paper describes a common workaround for speech: an external automatic speech recognition system transcribes the command into text for a text-only policy. The authors name two issues with that cascade. It makes the robot system larger and more complex, and transcription discards information beyond the words, such as identity, emotion and intonation.

Their example is the instruction “Please pick up my cup”, which a text-instruction policy may not resolve to the right cup. VLAS instead processes the raw speech inside the policy model.

VLAS is built on the open-source vision-language model LLaVA. The paper uses CLIP as the visual encoder and Vicuna, a fine-tuned LLaMA variant, as the foundation model. A Whisper encoder converts speech into hidden states: the signal becomes an 80-bin mel-spectrogram padded to 3,000 frames, the encoder yields 1,500 hidden representations, and a reduction factor of 5 shortens the sequence. Separate MLP projectors map speech and vision tokens into the language space.

Actions are generated autoregressively as discretized tokens. Each continuous action value is split into 256 uniformly spaced bins, using the 256 least frequently used tokens in the vocabulary, for a 7-dimensional action covering end-effector position, rotation and gripper state.

Voice retrieval-augmented generation lets the model use knowledge about the speaker. A pre-trained speaker identification module extracts a voiceprint from the raw command. The voiceprint is the key to an external database, and the retrieved background knowledge is passed to the language model alongside the visual and speech tokens.

The authors use this for tasks such as object ownership and user preference, where the same words call for different actions depending on who speaks.

Training has three stages. Stage I aligns speech with text using LibriSpeech, updating only the MLP between the speech encoder and the language model. Stage II fine-tunes on a new Speech Question Answering (SQA) dataset, LLaVA’s visual question answering data and LibriSpeech, producing the VLAS-Base model. Stage III fine-tunes on a new CALVIN with Speech Instructions (CSI) dataset.

Both new datasets use text-to-speech. SQA has 185K samples with over 1,152 voices. For CSI the authors converted CALVIN’s 389 textual instructions with 500 voices into about 194K audio samples, and randomly replaced half of the training samples with speech instructions.

The authors evaluate on the CALVIN benchmark and on a customization benchmark they built in simulation, and also describe a real UR5 robot arm experiment fine-tuned on the Berkeley UR5 demonstration dataset and their own cup-picking dataset. On CALVIN they report that VLAS with speech instructions outperforms cascaded baselines that add an external speech recognition model, and is comparable to a text-instruction baseline.

These are the authors’ claims on their own setups, and most speech instructions in training and evaluation are synthesized. They also evaluated real speech recorded from 10 individuals. Firsthand does not reproduce any of it.

A vision-language-action model, as the site defines it, conditions on images and a language instruction and outputs actions. VLAS keeps that structure and adds speech as a second instruction modality. The paper’s appendix also compares against OpenVLA on CALVIN and attributes the gap partly to OpenVLA’s third-person-only input. It is related to the vision-language-model and action-tokenization ideas the site defines. Firsthand ranks none of these models.

This entry draws on the arXiv paper and the official GitHub repository. The paper says the model, data and code will be publicly available there, and the repository page lists the Apache License 2.0. Whether the released weights carry a separate licence is Not stated here. The sources do not state that human egocentric video, stereo depth, hand pose or IMU data was used, so how those would suit VLAS is Not stated.

When a program needs real human egocentric demonstrations of tasks, environments or objects that public data does not cover, specified in writing. Firsthand does not build or train models, and the language it delivers is one task instruction per episode. Spoken-instruction audio is not part of what the site states.

Confirm the spec on a Spec Pilot, a small fast first batch. Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks. Billing is per validated hour.

What is VLAS?

A published vision-language-action model (arXiv 2502.13508, ICLR 2025) that integrates speech recognition into the robot policy, so spoken commands guide actions without a separate speech recognition system.

What is Voice RAG in VLAS?

A retrieval module that extracts a voiceprint from the raw command, uses it to query an external database, and passes the retrieved knowledge to the model for personalized tasks such as object ownership.

What is VLAS built on?

The paper builds it on LLaVA with a CLIP visual encoder, Vicuna as the foundation model and a Whisper encoder for speech, with actions output as discretized tokens.

Is the code open?

The paper says the model, data and code will be publicly available on its GitHub repository, whose page lists the Apache License 2.0. A separate weights licence is Not stated in the sources read.

Does Firsthand data train VLAS?

No claim is made. Firsthand is not affiliated with its authors, and the site does not state that any model was trained on Firsthand data.

Does Firsthand collect spoken-instruction data?

The site makes no speech or audio-instruction capture claim. It states custom egocentric collection to a written spec with one task instruction per episode, confirmed on a Spec Pilot.