What Is MolmoAct?

MolmoAct is introduced in MolmoAct: Action Reasoning Models that can Reason in Space (arXiv 2508.07917) by Jason Lee, Jiafei Duan, Haoquan Fang and co-authors at the Allen Institute for AI and the University of Washington. The paper starts from the point that most robotic foundation models map perception and instructions directly to control, which limits adaptability, generalization and semantic grounding. It introduces Action Reasoning Models (ARMs), a class of models that integrate perception, planning and control through a structured three-stage pipeline, and presents MolmoAct as the first such model. The authors describe it as fully open: model weights, training code and datasets are released.

MolmoAct is third-party research. Firsthand is not part of it or affiliated with its authors, and makes no claim that any model was trained on Firsthand data. What the site states is custom egocentric collection of real human demonstrations, fast and on demand, to a written spec: 3D hand pose at 21 joints per hand, calibrated stereo depth and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset with a buyer-owned commercial license. The only language data the site states is one task instruction per episode. Whether Firsthand data would suit a MolmoAct-style pipeline is Not stated in the sources read.

Per the paper, vision-language-action models aim to bring the flexibility of large language and vision models to robots, but remain brittle and opaque. They struggle to transfer across tasks, scenes and embodiments, and offer little insight into why a robot chose one action over another.

The authors argue the gap is partly a lack of structure. Prior work that adds reasoning to VLAs mostly reasons in language, for example by splitting a task into sub-tasks. The paper says this ignores two things precise control needs: depth perception and precise motion planning. MolmoAct is built to reason in space instead.

Conditioned on camera images and a language instruction, the model generates three token sequences in order. First come depth perception tokens, which the authors describe as a structured 2.5D representation of the scene. Second comes a visual reasoning trace, a short polyline of up to five points in image space that sketches where the end-effector will move. Third come the action tokens, the low-level robot command.

Each stage is conditioned on the one before it, so the final action is grounded in the inferred depth and the planned path. The authors note that each stage can be decoded on its own: a depth map, a 2D trajectory overlay on the image, and the executed action. The paper also says MolmoAct does not use intermediate reasoning in language.

The paper says the depth tokens are predicted from the RGB observation, not read from a depth sensor. A specialist depth estimator, built on Depth Anything V2, produces ground-truth depth maps. A VQVAE then quantizes each depth map into a fixed-length string of discrete tokens, and the VLA is trained to predict that string from the RGB image. The authors call this specialist-to-generalist distillation.

The labels for the trace are generated in a similar way. The authors prompt Molmo to point to the robot gripper in each frame, then link the points across the episode. For bimanual robots the left and right grippers are prompted separately.

Because the plan is a visible trace on the image, a user can draw or edit the path directly, and the model takes the trace-overlaid image as input when it predicts the next action. The paper says language-only steering is ambiguous about magnitudes, scales and endpoints, and describes trace steering as more precise. The Ai2 blog describes users sketching paths on a phone, tablet or laptop.

These are the authors’ design claims and results on their own setups. Firsthand does not test steering and quotes no scores.

Training has pre-training, mid-training and post-training stages. Pre-training uses a subset of the Open X-Embodiment mixture, named in the paper as BC-Z, BridgeData V2 and RT-1, converted into action reasoning data, plus auxiliary depth and trace data and multimodal web data. Mid-training uses the MolmoAct Dataset, which the authors collected in-house. Post-training adapts the model to a specific task and embodiment.

The abstract describes the MolmoAct Dataset as a mid-training robot dataset of over 10,000 high-quality trajectories across diverse scenarios and tasks. The paper says it was recorded with a single-arm Franka robot in home and tabletop settings, with two side camera views and one wrist camera view. Firsthand quotes no further counts from these sources and does not rank or endorse any dataset.

MolmoAct builds on Molmo, Ai2’s open vision-language model, reusing its vision encoder, connector and language model. The paper trains two variants: MolmoAct-7B-D, on a SigLIP2 and Qwen2.5-7B backbone, and MolmoAct-7B-O, on an OpenCLIP and OLMo2-7B backbone, which the authors call the most open. Actions are discretized into 256 bins per dimension and predicted as tokens.

The paper reports evaluations on SimplerEnv, on LIBERO, and in real-world fine-tuning on single-arm and bimanual robots, plus out-of-distribution generalization and human-preference tests for instruction following and trajectory steering. The authors state that MolmoAct outperformed the baselines they compared against.

These are the authors’ results on their own setups and baselines. Firsthand does not reproduce them and quotes no scores.

The official repository, allenai/molmoact, is released under the Apache License 2.0. Its README lists the model checkpoints, the MolmoAct Dataset in LeRobot format, and the pre-training and mid-training mixtures, along with code for data processing, fine-tuning, inference and evaluation. The data-processing step converts a LeRobot-format dataset into action reasoning data, using Depth Anything V2 and the authors’ VQVAE.

The README’s update log also lists a later release called MolmoAct2. This entry covers MolmoAct as described in the paper and does not cover MolmoAct2.

The site’s embodied chain-of-thought entry covers a method that writes out reasoning in text, such as a plan, sub-task and object bounding boxes, before each action. The site’s CoT-VLA entry covers a model that predicts a future subgoal image before acting. MolmoAct instead reasons through depth tokens and a 2D end-effector trace, and the paper says it avoids reasoning in language.

The site’s SpatialVLA entry covers a model that adds 3D position encoding and spatial action tokens, and the MolmoAct paper compares against it. All of these are third-party research, and Firsthand ranks none of them.

This entry draws on the arXiv paper, the official GitHub repository and the official Ai2 blog post. They do not state how calibrated stereo depth, 3D hand pose or IMU streams would be used in MolmoAct, or whether human egocentric video is an input. The sources describe robot trajectories, with the gripper located in the image, so using human hand pose as a source for the trace is Not stated.

When a program needs real human egocentric demonstrations of tasks, environments or objects that public data does not cover, specified in writing. Firsthand does not build or train models, and the language it delivers is one task instruction per episode.

Confirm the spec on a Spec Pilot, a small fast first batch. Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks. Billing is per validated hour.

What is MolmoAct?

A published open Action Reasoning Model (arXiv 2508.07917) that predicts depth perception tokens, then a 2D visual reasoning trace, then robot action tokens.

What is an Action Reasoning Model?

The authors’ term for a robotic foundation model that integrates perception, planning and control through a structured three-stage reasoning pipeline, instead of mapping inputs directly to actions.

Does MolmoAct use a depth sensor?

Per the paper, no. Depth tokens are predicted from the RGB observation, trained against depth maps from a specialist estimator.

Is the MolmoAct code and data available?

The official allenai/molmoact repository is Apache 2.0 licensed and lists model checkpoints, the MolmoAct Dataset in LeRobot format, training mixtures and code for data processing, fine-tuning and evaluation.

Does Firsthand data train MolmoAct?

No claim is made. Firsthand is not affiliated with the authors, and the site does not state that any model was trained on Firsthand data.

Can Firsthand collect egocentric data for a VLA program?

The site states custom egocentric collection to a written spec, fast and on demand, with 21-joint 3D hand pose, calibrated stereo depth, 200 Hz IMUs and one task instruction per episode, confirmed on a Spec Pilot.