What Is SpatialVLA?

SpatialVLA is introduced in SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model (arXiv 2501.15830) by Delin Qu, Haoming Song, Qizhi Chen and co-authors, listed on arXiv as Robotics: Science and Systems 2025, with an official project page, a GitHub repository and Hugging Face checkpoints. The abstract claims that spatial understanding is the keypoint in robot manipulation, and proposes to explore effective spatial representations for a robot foundation model. Its two proposals are Ego3D Position Encoding for the input observations and Adaptive Action Grids for the actions. This page is a definition, not a guide to running the code.

SpatialVLA is third-party research. Firsthand is not affiliated with its authors or with Shanghai AI Laboratory, and makes no claim that SpatialVLA or any other named model was trained, fine-tuned or evaluated on Firsthand data. Firsthand captures first-person human video, collected fast, on demand and to a written spec, not robot or teleoperation data, and it does not deliver robot actions. The raw fields the site states are depth derived from a calibrated stereo rig, 3D hand pose at 21 keypoints per hand with a per-joint visibility flag, action segments, and 6-DoF IMUs at 200 Hz on one clock. The only language field is one task instruction per episode. Whether those fields suit a model of this kind is Not stated in the sources read.

According to the abstract, Ego3D Position Encoding injects 3D information into the input observations of the visual-language-action model. The project page calls the same idea 3D Egocentric Position Encoding. How the 3D information is computed and encoded is Not stated in the sources read.

Adaptive Action Grids represent spatial robot movement actions with adaptive discretized action grids. The abstract says this helps the model learn spatial action knowledge that transfers across robots. After pre-training, the pre-learned grids can be re-discretized to capture the robot-specific spatial action movements of a new setup, which is how the authors fine-tune for new simulation and real-world setups.

The project page describes a pipeline that takes an image observation and a natural-language task instruction. The model processes the image with Ego3D Position Encoding, auto-regressively predicts spatial action tokens, and de-tokenizes them into continuous actions for robot control. It lists three key components: a SigLIP vision encoder that extracts 2D semantic features while Ego3D Position Encoding injects 3D spatial context, unified adaptive action grids that discretize spatial action movements into separate spatial action tokens, and grids and tokens that can adjust and update for new robot setups.

The GitHub README says the vision-language backbone is PaliGemma2, and the Hugging Face card describes the released checkpoint as fine-tuned from paligemma2-3b-pt-224 with a model size of 4B parameters. The card lists the model type as language and image in, robot actions out, and its usage example passes one image and a text prompt. The README also says an advanced version supporting multi-view and state inputs is under development. That is a statement about a future version, not the released model.

On the project page, the 2D semantic features come from the SigLIP vision encoder, and Ego3D Position Encoding is the step that adds 3D spatial context on top. The authors credit that encoding for SpatialVLA’s performance in understanding spatial prompts and complex spatial layouts. They compare it with Diffusion Policy, Octo, OpenVLA, RT-1-X and RoboVLM, and report 73% accuracy on one Franka task that involves spatial prompts. That is the authors’ own result on their own evaluation.

The project page concludes by suggesting that 3D information, including depth or point cloud, be integrated into the VLA framework to improve adaptability and robustness to spatial layout changes. It does not say which 3D source its own encoding uses, so this page does not either. The comparison table below sets out only what the sources state.

The abstract says SpatialVLA was first pre-trained on top of a vision-language model with 1.1 million real-world robot episodes, then applied zero-shot to many tasks. The GitHub README and the Hugging Face card say those demonstrations come from the Open X-Embodiment and RH20T datasets, and that pre-training ran on a cluster of 64 A100 GPUs for about 10 days with a batch size of 2048. Most fine-tuning experiments used LoRA on 4 or 8 A100 GPUs.

The project page says the evaluation covers 7 robot learning scenarios, 16 real-world robot tasks and 48 simulation setups. The Hugging Face card also states a limit: the models do not zero-shot generalize to unseen robot embodiments or setups missing from the pre-training mix, and it suggests collecting demonstrations on the desired setup and fine-tuning instead. Both the checkpoints and the training codebase are released under the MIT license, per the card.

A model with this input and output needs episodes that pair an image observation, an instruction and a time-aligned action signal in the robot’s action space. For custom data, the README recommends converting the dataset to RLDS and then editing the dataset mixture, configuration and transform files in the repository. Firsthand does not deliver robot action trajectories, and the site states no mapping from its fields to an action-grid format. The last table sets those needs against the fields the site states.

If your program needs different fields, signals or environments, collection is custom, fast and on demand, scoped by a written spec. Settle the fields, the delivery format, the depth encoding and the wording of the single task instruction per episode on a Spec Pilot, a small first batch, then request a custom dataset.

What does SpatialVLA stand for?

The paper title describes exploring spatial representations for a visual-language-action model. The sources read do not spell the name out further.

How does SpatialVLA output actions?

Per the project page, it auto-regressively predicts spatial action tokens, defined by unified adaptive action grids, and de-tokenizes them into continuous actions for robot control.

Does SpatialVLA need a depth sensor?

Not stated in the sources read. The project page says Ego3D Position Encoding injects 3D spatial context, but not which 3D source it uses, and the Hugging Face usage example passes one image and a text prompt.

Is SpatialVLA open source?

The Hugging Face card states that all SpatialVLA checkpoints and the training codebase are released under the MIT license.

Does Firsthand deliver data in SpatialVLA format?

No. Firsthand delivers raw fields from first-person human capture, such as calibrated stereo-derived depth, 3D hand pose, action segments and IMU, and does not ship robot actions. Mapping them to action grids is Not stated and stays in your pipeline.

What language data does Firsthand provide?

One task instruction per episode, paired with the video and sensor streams. The site states no narration or per-step language labels.