What Is villa-X?

villa-X is introduced in villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Models (arXiv 2507.23682) by Xiaoyu Chen, Hangxing Wei, Pushi Zhang, Chuheng Zhang, Kaixin Wang, Yanjiang Guo, Rushuai Yang, Yucen Wang, Xinquan Xiao, Li Zhao, Jianyu Chen and Jiang Bian. The paper starts from latent action learning, where a Latent Action Model (LAM) compresses the motion between successive frames into compact latent tokens that act as pseudo-action labels, so that action-free video can feed robot policy training. It asks two questions: how to learn better latent actions, and how to integrate them more effectively into VLA pretraining. villa-X answers both, and the authors call the result a Vision-Language-Latent-Action (ViLLA) framework.

villa-X is third-party research. Firsthand is not part of it or affiliated with its authors, and makes no claim that any model was trained on Firsthand data. What the site states is custom egocentric collection of real human demonstrations, fast and on demand, to a written spec: 3D hand pose at 21 joints per hand, calibrated stereo depth and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset with a buyer-owned commercial license. The site’s latent-actions entry notes that Firsthand’s depth, hand pose and action-segment data are captured and annotated at the time of recording, not inferred afterwards. The only language data the site states is one task instruction per episode. Whether Firsthand data would suit a villa-X-style pipeline is Not stated in the sources read.

Per the paper, latent action models usually compress changes in the visual signal. That tracks robot motion in general, but some motions, such as end-effector rotations or gripper movements, are subtle in pixel changes yet critical for control. A vision-only latent action can pay little attention to them, so the learned latent actions stay physically ungrounded and transfer less well.

villa-X targets that gap. It keeps the idea of learning from action-free video, and changes what the latent action is trained to explain.

The LAM has an inverse dynamics model that infers a latent action from a pair of observations, and a visual forward dynamics model that reconstructs the later frame from the current frame and the latent action. This is the standard recipe.

villa-X adds a proprioceptive forward dynamics model (proprio FDM). It predicts future robot states and actions from the latent action, the current robot state and an embodiment context, so the latent token has to carry information that matches the robot’s physical dynamics. The embodiment context is built from a dataset ID and the control frequency, which helps separate embodiments in mixed data. For human video, which has no robot state labels, the proprio term is left out and only the visual objective applies.

The second component, called ACT, sits on a pre-trained vision-language model. The paper’s appendix names PaliGemma as that backbone. Two further experts hang off it: ACT-latent predicts a sequence of latent actions as a mid-level plan, and ACT-robot predicts the low-level robot action chunk, conditioned on the VLM features, the predicted latents, the proprioceptive state and the embodiment context.

The two are modelled together with a joint diffusion process, so robot actions are conditioned on latent actions during training and not only through pre-trained weights. The authors also stochastically mask attention from robot actions to latent actions, to stop the robot branch relying too heavily on them, and describe this as crucial in practice. The low-level policy can optionally take wrist camera input when it is available.

Training runs in three stages: LAM pretraining on diverse datasets, ACT pretraining with joint latent and robot modelling, and embodiment-specific finetuning. The project page says both LAM and ACT are trained on a large collection of robotic data and human egocentric videos, including Open X-Embodiment, Something-Something V2, Ego4D and others.

The appendix adds that LAM pretraining uses only the primary third-person camera view. Egocentric video is therefore one input to a mixed pretraining pool in this method. Firsthand does not rank or endorse any of those datasets and quotes no counts from them.

The paper reports state-of-the-art performance in numerous simulation tasks in SIMPLER and in two real-world setups. The real-world platforms are a Realman arm with a gripper and an XArm with a 12-DoF XHand dexterous hand. The authors also evaluate on the LIBERO suites, and show that the latent action expert generalizes zero-shot to an unseen embodiment and to open-vocabulary symbolic icons.

These are the authors’ results on their own setups and baselines. Firsthand does not reproduce them and quotes no scores.

The official repository, microsoft/villa-x, is released under the MIT License. Its README lists a pre-trained Latent Action Model and LAM inference code, with a notebook walking through extracting latent actions from a video and reconstructing future frames. Its release plan marks the Latent Action Model as released and the Actor Model as not yet released at the time this entry was written, so check the repository for the current state.

The site’s LAPA entry covers a method that learns discrete latent actions from video and pretrains a model to predict them. The paper itself says villa-X offers tighter integration than LAPA, because latent actions are modelled jointly with robot actions and not used only as a pretraining target. The paper describes UniVLA as a two-stage pipeline that learns task-centric latent actions.

The site’s UniAct entry covers a universal action space shared across embodiments. villa-X’s distinguishing parts, as stated by its authors, are the proprio FDM grounding and the joint latent-and-robot action diffusion. All of these are third-party research, and Firsthand ranks none of them.

This entry draws on the arXiv paper, the official GitHub repository and the official project page. They do not state how stereo depth, 3D hand pose or IMU streams would be used in villa-X. The paper does say the framework is general and could use cues such as end-effector keypoint detection or human hand pose estimation in place of low-level robot states, and leaves that for future work, so it is not an evaluated result.

When a program needs real human egocentric demonstrations of tasks, environments or objects that public data does not cover, specified in writing. Firsthand does not build or train models, and the language it delivers is one task instruction per episode.

Confirm the spec on a Spec Pilot, a small fast first batch. Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks. Billing is per validated hour.

What is villa-X?

A published Vision-Language-Latent-Action framework (arXiv 2507.23682) that improves how latent actions are learned and how they are used in VLA pretraining.

What are latent actions in villa-X?

Compact latent tokens that capture the motion between successive frames and act as pseudo-action labels. villa-X grounds them by also predicting future robot states and actions through a proprioceptive forward dynamics model.

How does villa-X differ from LAPA?

The authors say it integrates latent actions more tightly, by modelling latent actions and robot actions together with a joint diffusion process, and by adding proprioceptive grounding to the latent action model.

Is the villa-X code available?

The official microsoft/villa-x repository is MIT licensed and lists a released pre-trained Latent Action Model with inference code. Its release plan showed the Actor Model as not yet released when this entry was written.

Does Firsthand data train villa-X?

No claim is made. Firsthand is not affiliated with the authors, and the site does not state that any model was trained on Firsthand data.

Can Firsthand collect egocentric data for a VLA program?

The site states custom egocentric collection to a written spec, fast and on demand, with 21-joint 3D hand pose, calibrated stereo depth, 200 Hz IMUs and one task instruction per episode, confirmed on a Spec Pilot.