What Is EgoVLA?

EgoVLA is introduced in EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videos (arXiv 2507.12440) by Ruihan Yang, Qinxi Yu and colleagues at UC San Diego, MIT, NVIDIA and UIUC, with an official project page and GitHub repository. The abstract starts from a data limit: collecting real robot data needs robot hardware, which constrains how much can be collected. The authors say human video offers scale and, more importantly, richness of scenes and tasks. Their vision-language-action model is trained on human video to predict human wrist and hand actions. Inverse kinematics and retargeting then convert those actions into robot actions, and a few robot manipulation demonstrations fine-tune the model into the robot policy they call EgoVLA. The authors also propose a simulation benchmark, the Ego Humanoid Manipulation Benchmark. The paper’s supplementary material and its later results tables were not read for this entry.

EgoVLA is third-party research. Firsthand is not affiliated with its authors or their institutions, makes no claim that EgoVLA or any named model was trained on Firsthand data, and makes no claim that Firsthand data is compatible with or used by EgoVLA. Firsthand captures first-person human video, collected fast, on demand and to a written spec. It is not robot or teleoperation data. The fields the site states are head and wrist camera video, depth derived from a calibrated stereo rig, 3D hand pose at 21 keypoints per hand with a per-joint visibility flag, wrist trajectories, action segments, and 6-DoF IMUs at 200 Hz on one clock. The only language field is one task instruction per episode.

EgoVLA takes visual history, a language instruction, action query tokens and the human’s current hand state. Per the paper, the visual input is six RGB frames: the current frame and five earlier ones, sampled 0.2 seconds apart to cover one second, each 384 by 384. The instruction describes the immediate desired behaviour. Proprioception means wrist translation and rotation plus hand pose parameters, passed through MLPs.

A vision-language model, NVILA-2B, encodes these inputs. A transformer action head of 300M parameters, with six encoder layers, predicts future actions over a one-second horizon of 30 steps at 30 Hz for both hands. Each action is a wrist pose (3D translation and rot6D rotation in the camera frame) plus hand joint angles, given as the top 15 PCA components of the MANO hand model. The paper says full loss details are in its supplementary material, so the loss weights are Not stated in the sources read.

The paper’s key observation is that human and robot action spaces differ by a few geometric transforms. A unified action space uses wrist pose and MANO hand parameters for both. To fine-tune on robot data, the authors align the robot to the human representation: end-effector poses through 3D transformations, and robot hand configurations by fitting MANO parameters whose fingertip positions best match the robot hand’s.

At inference, predicted wrist poses become robot end-effector poses and arm joint angles come from inverse kinematics. MANO gives 3D hand keypoints, and a lightweight MLP maps them to robot hand joint commands. The paper reports a mean fingertip position error of 5×10⁻⁵ m for that mapping. Direct transfer is hard, the paper says, because of differences in camera pose, hand morphology and visual appearance, so it fine-tunes on a small set of robot demonstrations after human pretraining.

Per the paper, the human dataset combines four sources and holds egocentric RGB, wrist poses, hand poses and camera poses. Observations are sampled at 3 FPS, giving about 500,000 image-action pairs, and world-frame camera poses project future wrist positions into the current camera frame. The README says these human datasets supply egocentric RGB video, hand, head and camera pose, and language labels.

For robots, the benchmark runs in Isaac Lab with a Unitree H1 humanoid and two Inspire dexterous hands. It has 12 tasks, 7 short-horizon and 5 long-horizon, with 100 demonstrations per task collected with OpenTelevision on a Meta Quest 3 and 25 visual background combinations. The paper says it is not designed for direct sim-to-real transfer. Real-robot results are Not stated in the sources read.

On human motion, the paper reports an average future wrist translation error of about 8 cm and a normalized 2D image error of about 0.13. On short-horizon benchmark tasks, Table 1 reports mean success rates on seen visual configurations of 77.78 for EgoVLA, 64.55 for EgoVLA-NoPretrain and 24.87 for ACT, and on unseen configurations 69.11, 51.28 and 24.89.

The project page says pretraining on egocentric human video boosts in-domain and out-of-domain performance, and that more diverse human data mixtures improve generalization. These are simulation results on the authors’ own benchmark. Ablation numbers are Not stated in the sources read.

The paper contrasts EgoVLA with VLAs that fine-tune vision-language models on large-scale robot data, noting that such training is data-intensive and often needs teleoperation or scripted execution. It names OpenVLA and Octo as using crowd-sourced robot datasets. EgoVLA instead pretrains on human video and uses robot demonstrations only to fine-tune.

EgoMimic, covered in its own entry, co-trains one policy on human and robot data together. Whether the EgoVLA paper compares against EgoMimic is Not stated in the sources read, and this page ranks neither.

The training signal in the paper is wrist and hand motion in the camera frame, plus an instruction for the behaviour wanted. That maps onto what a collection spec can name: tasks, environments, hand and wrist labels and an instruction per episode.

Firsthand states 3D hand pose at 21 keypoints per hand in metric units in the head-camera frame, aligned to depth and RGB, with a per-joint visibility flag, plus wrist trajectories and action segments. MANO parameters, rot6D wrist poses and PCA hand features are Not stated as Firsthand deliverables, and nothing here says Firsthand data is compatible with EgoVLA. EgoVLA’s instruction describes the immediate behaviour, while the site’s only language field is one task instruction per episode, so whether that fits is for your team to check. Converting a delivered dataset to a model’s action space is a step in your own pipeline.

EgoVLA is a model and a benchmark, not a data source, and its code, checkpoints and simulation data come from its authors. When a program needs human video from particular tasks, environments or people, write them into a spec and confirm it on a Spec Pilot, a small fast first batch. Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks. Billing is per validated hour, and delivery is in the format your training stack already reads: RLDS, LeRobot, WebDataset or HDF5. Firsthand does not deliver robot or teleoperation data, so the robot demonstrations a model like this fine-tunes on come from your own setup.

What is EgoVLA?

A vision-language-action model from a July 2025 paper. It is trained on egocentric human video to predict future wrist and hand actions, which inverse kinematics and retargeting convert to robot actions, then fine-tuned on a few robot demonstrations.

What does EgoVLA predict?

Per the paper, future wrist poses (3D translation and rot6D rotation in the camera frame) and hand joint angles, given as the top 15 PCA components of the MANO hand model, over a one-second horizon of 30 steps at 30 Hz for both hands.

What data was EgoVLA trained on?

Human pretraining used a combined dataset from HOI4D, HOT3D, HoloAssist and TACO, about 500,000 image-action pairs in total. Robot fine-tuning used simulated humanoid demonstrations from the authors’ benchmark. Real-robot data is Not stated in the sources read.

Is EgoVLA open source?

The authors release training and evaluation code on GitHub and checkpoints and simulation data on Hugging Face. The repository lists its licence as Other, the dataset page lists MIT, and the model page has no model card. The README notes the code has not been fully tested. Read each licence yourself; this is not legal advice.

Does EgoVLA take language instructions?

Yes. Per the paper, the input includes a language instruction describing the immediate desired behaviour. How instructions are written for each dataset is Not stated in the sources read beyond the README noting language labels.

Does Firsthand use EgoVLA, and can I request a custom dataset?

Firsthand is not affiliated with the EgoVLA authors, makes no claim that any model was trained on Firsthand data, and makes no claim that its data is compatible with or used by EgoVLA. You can request a custom egocentric dataset: write the tasks, fields and format into a spec, confirm it on a Spec Pilot, and billing is per validated hour. The only language field is one task instruction per episode.