What Is VLA-OS?

VLA-OS is introduced in VLA-OS: Structuring and Dissecting Planning Representations and Paradigms in Vision-Language-Action Models (arXiv 2506.17561), posted in June 2025 by Chongkai Gao, Lin Shao and co-authors from the National University of Singapore and partner institutions. The authors note that existing planning-based VLAs differ in architecture, paradigm, representation and training data, so it is hard to see where gains come from. VLA-OS isolates the planning choice. The “OS” stands for “Operating System” in the paper, meaning unified interfaces for planning heads and paradigms.

VLA-OS is third-party research. Firsthand is not part of it or affiliated with its authors, and makes no claim that any model was trained on Firsthand data. What the site states is custom egocentric collection of real human demonstrations, fast and on demand, to a written spec: 3D hand pose at 21 joints per hand, calibrated stereo depth and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset with a buyer-owned commercial license. The only language data the site states is one task instruction per episode. Firsthand does not deliver plan, subtask or motion-reasoning labels.

The paper says recent VLAs have moved from end-to-end action generation toward a pipeline of task planning followed by action generation. These methods vary in network design, planning paradigm, representation and training data, so researchers cannot tell which component helps. The authors ask five questions: which representation to use, whether one model or two should plan and act, which part is the bottleneck, whether planning keeps scaling and pretraining benefits, and whether it helps generalization and continual learning.

The paper names four. PlanningOnly-VLA plans without producing low-level actions and is left out because such systems usually cannot be trained on the given datasets. ActionOnly-VLA maps images and language directly to actions. Integrated-VLA uses one model for planning and acting, either explicitly, where actions condition on the plan, or implicitly, where planning is only an auxiliary training loss.

Hierarchical-VLA uses two separate models with no connection or gradient between them. The VLA-OS models are VLA-OS-A, VLA-OS-I (with implicit and explicit variants) and VLA-OS-H.

Three. Language reasoning has eight keys per timestep: Task, Plan, Subtask, Subtask Reason, Move, Move Reason, Gripper Position and Object Bounding Boxes. Visual reasoning uses location tokens for object bounding boxes, end-effector flow and target-object affordance. Image foresight is a future third-person image used as a short-horizon goal.

The authors annotate these with vision-language models, such as Prismatic-7B and Gemini 1.5, and then refine the output to keep plans consistent across episodes of a task.

Each model pairs a vision-language backbone with an action head and optional planning heads. The backbone uses Qwen2.5 language models from 0.5B to 7B parameters with DINOv2 and SigLIP vision encoders. The authors cap the heads at about 5% of the backbone’s parameters to keep comparisons fair. Their code supports other Hugging Face language models.

The abstract reports two headline results: visually grounded planning representations are generally better than language ones, and Hierarchical-VLA generally matches or beats the other paradigms, at the cost of slower training and inference. The paper also reports that, on its LIBERO tasks, policy learning is harder than task planning, and that explicit planning in Integrated-VLA hurt when trained from scratch.

These are the authors’ claims on their own setups, and Firsthand does not reproduce them. The main figures appear in the results table below, attributed to the authors.

VLA-OS is a comparison study, not a single method. The paper places embodied chain-of-thought among Integrated-VLA methods and Hi Robot among Hierarchical-VLA methods, and it cites RoboBrain as an Integrated-VLA example with auxiliary reasoning loss.

The site’s task decomposition entry is a different level of the same idea: splitting a human job into task units for capture. Firsthand ranks none of these.

The experiments use LIBERO, The Colosseum, FurnitureBench, DexArt and PerAct2, plus real-world deformable-object tasks that the authors collected by human teleoperation with 100 demonstrations per task. These are robot demonstrations, in simulation and on real arms, each with a language instruction. The paper does not state that human egocentric video, stereo depth, hand pose or IMU data was used.

The authors list three limits: some planning designs are not covered, such as latent actions, video generation and scene flow; embodiment, sim-to-real and 2D-to-3D transfer were not explored; and their training data stays under 10,000 trajectories, with no pretraining on larger sets such as Open X-Embodiment.

The GitHub README lists MIT licensing for code, model weights and data, and notes that evaluation code and continual-learning training code were still to come at review time. The model card says only Qwen2.5 VLM checkpoints and training networks were released so far. How VLA-OS would use egocentric human data is Not stated.

When a program needs real human egocentric demonstrations of tasks, environments or objects that public data does not cover, specified in writing. Firsthand does not build or train models. Its language data is one task instruction per episode.

Confirm the spec on a Spec Pilot, a small fast first batch. Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks. Billing is per validated hour.

What is VLA-OS?

A published family of vision-language-action models and controlled experiments (arXiv 2506.17561) that compares planning paradigms and planning representations while holding architecture and training data fixed.

What is the difference between integrated and hierarchical VLA?

Per the paper, Integrated-VLA uses one model for planning and acting, while Hierarchical-VLA uses two separate models with no connection or gradient between them.

Which planning representation did the authors favor?

The abstract reports that visually grounded planning representations are generally better than language planning representations. These are the authors’ results on their own setups.

Is the code open?

The official GitHub repository lists the MIT License, and the README says code, model weights and data are MIT licensed. The model card says only some checkpoints were released so far.

Does Firsthand data train VLA-OS?

No claim is made. Firsthand is not affiliated with its authors, and the site does not state that any model was trained on Firsthand data.

Does Firsthand deliver planning labels?

The only language data the site states is one task instruction per episode. Plan, subtask and motion-reasoning labels are Not stated. Scope is confirmed on a Spec Pilot.