Glossary
What Is 3D-VLA?
Last reviewed: October 2026
Short answer
3D-VLA is a 3D vision-language-action generative world model from researchers at UMass Amherst and partner institutions, introduced in arXiv 2403.09631. It builds on a 3D-based large language model with interaction tokens, and aligned diffusion models predict goal images and point clouds that guide robot control. Firsthand is not part of it.
01
What is 3D-VLA?
3D-VLA is introduced in 3D-VLA: A 3D Vision-Language-Action Generative World Model (arXiv 2403.09631) by Haoyu Zhen, Chuang Gan and co-authors; the official GitHub repository lists ICML 2024. The abstract says recent VLA models rely on 2D inputs and map perception directly to action, neglecting world dynamics, and proposes embodied foundation models that link 3D perception, reasoning and action through a generative world model.
02
Where does Firsthand fit next to 3D-VLA?
3D-VLA is third-party research. Firsthand is not affiliated with its authors and claims no use, training or compatibility with it. Firsthand captures first-person human video, not robot data, fast, on demand and to a written spec: stereo-derived depth on the stereo capture pages, 21 keypoints per hand with a visibility flag, wrist trajectories, action segments and one task instruction per episode.
03
How is 3D-VLA built?
Per the paper, 3D-VLA follows the 3D-LLM method but starts from BLIP2-FlanT5 XL, unfreezing token embeddings and Q-Former weights. New interaction tokens mark objects, locations (six tokens per 3D bounding box), static scenes and 7-DoF robot actions.
04
How does 3D-VLA generate goal images and point clouds, as stated in the sources?
Two embodied diffusion models are pretrained: RGB-D to RGB-D from Stable Diffusion V1.4, and point cloud to point cloud from Point-E. A transformer-based projector aligns them with the LLM. The generated goal state is fed back to guide robot control. The README says the diffusion models are on Hugging Face.
05
What data was 3D-VLA trained on, as stated in the sources?
The paper curates 2M 3D-language-action pairs from existing data, including 12 Open X-Embodiment datasets, the RLBench and CALVIN simulators, and human-object interaction datasets such as Epic-Kitchens and HOI4D. Over 95% of embodied video datasets lacked 3D information, so depth was estimated with ZoeDepth and lifted to point clouds. Hours of video are Not stated in the sources read.
06
Why do 3D world models matter for robot manipulation?
The authors argue that people plan by imagining future 3D states, while earlier VLAs map perception straight to action. On CALVIN, five tasks in a row, the first task scores 44.7 against 28.2 for MCIL; on RLBench, Put Knife scores 68 against 28.8 for LanCon-Learn, though the paper notes that baseline uses history and 3D-VLA runs open loop. These are the authors’ own results; real-robot results are Not stated in the sources read. Firsthand draws no comparison.
07
How does 3D-VLA differ from 2D VLAs, SpatialVLA and world models?
Per the paper, earlier vision-language models with action output mostly used 2D features; 3D-VLA predicts future 3D features guided by the goal, and the authors claim to be first to use point clouds for action token generation. The SpatialVLA entry states Ego3D Position Encoding and Adaptive Action Grids. A world model, per the site entry, predicts how a scene changes in response to an action; 3D-VLA overlaps with one but also outputs action tokens. Any comparison between the papers is Not stated in the sources read.
08
What real-world egocentric data with depth supports 3D-aware policies, and how do you get it custom-collected?
Most embodied video has no 3D, so the authors estimated depth. Firsthand captures human first-person video, not robot data. On the stereo capture pages, depth is derived from a calibrated stereo rig in metric millimetres (HDF5 streams/depth_mm, RLDS observation.depth_mm, WebDataset depth.png). The site states no point clouds, 3D bounding boxes or robot actions. Confirm a written spec on a Spec Pilot; median time to first validated batch is 48 hours where coverage is deep, three to four weeks where thin. Mapping to 3D-VLA is the buyer’s pipeline.
09
3D-VLA inputs, outputs and training setup, as stated in the official sources
| Item | What the official sources state |
|---|---|
| Model type | Generative world model on a 3D-based LLM, linking 3D perception, reasoning and action |
| Inputs | Language instruction and 3D scene features |
| Outputs | Text, bounding-box tokens, 7-DoF action tokens, goal images, depths and point clouds |
| Goal diffusion | RGB-D from Stable Diffusion V1.4; point clouds from Point-E |
| Training data | 2M 3D-language-action pairs (paper) |
| Firsthand relationship | None |
11
3D-VLA beside SpatialVLA and a world model, as each official source or site entry states
| Item | 3D-VLA (sources read) | SpatialVLA (site entry) | World model (site entry) |
|---|---|---|---|
| Core idea | Imagines goal images and point clouds, then plans actions | Adds Ego3D Position Encoding and Adaptive Action Grids | Predicts how a scene changes in response to an action |
| 3D handling | Multi-view 3D scene features; point clouds lifted from depth | Ego3D Position Encoding injects 3D information into the visual input | Depth grounds predicted change in scene geometry |
| Action output | Discrete 7-DoF action tokens | Spatial action tokens from adaptive grids | Not stated in the site entry |
| Training data | 2M 3D-language-action pairs | 1.1 million real-world robot episodes | Dense frames paired with actions |
12
What 3D-VLA consumes beside what the Firsthand site states
| Aspect | 3D-VLA (sources read) | What the Firsthand site states |
|---|---|---|
| Depth | Estimated per frame with ZoeDepth where missing | Derived from a calibrated stereo rig, metric millimetres, on the stereo capture pages |
| Point clouds | Scene input and generated goals | Not stated on the site |
| Actions | 7-DoF robot actions from source datasets | Robot actions not delivered; action segments, wrist trajectories, 21 keypoints per hand with visibility flag |
| Language | Templated prompts diversified by ChatGPT | One task instruction per episode |
| File layout | Not stated in the sources read | RLDS, LeRobot v2.1, WebDataset, HDF5 |
| Compatibility | Not claimed | Not claimed |
13
Frequently asked questions
- What is 3D-VLA?
- A 3D vision-language-action generative world model that predicts goal images and point clouds and outputs robot action tokens.
- How does 3D-VLA produce goal point clouds?
- A Point-E-based diffusion model with a point cloud condition, aligned to the LLM by a transformer-based projector.
- How much data was 3D-VLA trained on?
- The paper states 2M 3D-language-action pairs. Hours of video are Not stated in the sources read.
- Is 3D-VLA open source?
- The official GitHub repository releases goal-generation and pretraining code plus diffusion models on Hugging Face. Its licence is Not stated in the README read.
- How is 3D-VLA different from SpatialVLA?
- Per the sources read, 3D-VLA generates goal images and point clouds; SpatialVLA adds Ego3D Position Encoding and Adaptive Action Grids. Any comparison between them is Not stated.
- Does Firsthand use 3D-VLA, and can I request a custom dataset?
- Firsthand is not affiliated with the 3D-VLA authors and claims no use, training or compatibility. You can request a custom dataset, confirmed first on a Spec Pilot.