What Is RoboVLMs?

RoboVLMs is introduced in arXiv 2412.14058, “What Matters in Building Vision-Language-Action Models for Generalist Robots”, with an official project page and GitHub repository. The project page regards VLAs as policy models fine-tuned from pre-trained vision-language models. Read: the abstract, project page and README; not the full paper.

RoboVLMs is third-party research. Firsthand is not affiliated with its authors and claims no use, training or compatibility with it or any named model. Firsthand captures first-person human video, not robot data, fast, on demand and to a written spec: 21 keypoints per hand with a visibility flag, wrist trajectories and action segments, plus one task instruction per episode.

The abstract names three design choices: which backbone to select, how to formulate the VLA architecture, and when to add cross-embodiment data. The project page adds a fourth question, why prefer a VLA at all.

The project page sorts VLA structures on two axes: input, whether history is observable, and output, whether actions are continuous or discrete. The README lists Flamingo, Qwen, LLaVA, Uform-Gen, MoonDream, PaliGemma and KosMos2 as supported backbones, and Detokenizer, MLP, LSTM and GPT2 as action decoders, with diffusion marked ongoing.

The project page reports, from the authors’ own experiments, four findings. VLAs were the best choice for generalist robot policies on robustness, generalization and data efficiency. The best structure took multi-step history as input and continuous actions as output, with a policy head to integrate history. KosMos and PaliGemma were distinctively better backbones, which the authors hypothesise reflects sufficient vision-language pretraining. Cross-embodiment data worked better in pre-training, and co-training on in-domain data from other tasks helped.

Scores are not reproduced here, and Firsthand draws no comparison from them.

RoboVLMs is a study plus a framework, so it spans models rather than sitting beside one. The table gives only what each source or entry states; Firsthand ranks none.

The sources describe robot data: Open X-Embodiment, CALVIN and a real-world set the project page sizes at 20 tasks and 8K+ trajectories. The README’s custom sample holds head and gripper images, actions and text. Egocentric human video and hand pose are not described.

In the sources, the training data is robot data with actions. Firsthand captures human first-person video, not robot data, and states no robot actions. Write tasks, environments and signals into a spec and confirm them on a Spec Pilot. Median time to first validated batch: 48 hours where coverage is deep, three to four weeks where thin. Formats: RLDS, LeRobot v2.1, WebDataset, HDF5; mapping to RoboVLMs’ sample layout is the buyer’s pipeline.

What is RoboVLMs?

A family of vision-language-action models plus an open framework and design study, introduced in arXiv 2412.14058.

What does the RoboVLMs paper study?

Three design choices: which backbone to select, how to formulate the VLA architecture, and when to add cross-embodiment data.

Which backbone does RoboVLMs find works best?

The project page says KosMos and PaliGemma were distinctively better in the authors’ experiments, possibly from sufficient vision-language pretraining.

Is RoboVLMs open source?

The abstract says code, models, datasets and toolkits are open-sourced, and GitHub shows Apache License 2.0. Weight licences are Not stated in the sources read.

Does RoboVLMs use egocentric human video?

Not described in the sources read. They describe robot data with actions.

Does Firsthand use RoboVLMs, and can I request a custom dataset?

Firsthand is not affiliated with the RoboVLMs authors and claims no use, training or compatibility. You can request a custom dataset, confirmed first on a Spec Pilot.