What Is Magma?

Magma is introduced in Magma: A Foundation Model for Multimodal AI Agents (arXiv 2502.13130), a CVPR 2025 paper by thirteen authors from Microsoft Research, the University of Maryland, the University of Wisconsin-Madison, KAIST and the University of Washington. The authors observe that vision-language-action models are usually trained separately for the digital world, such as UI navigation, and the physical world, such as robot manipulation, and that most prioritize a task-specific action policy at the cost of generic multimodal understanding. Magma is their single model that keeps verbal understanding and adds the ability to plan and act in the visual-spatial world. The project page calls it the first foundation model for multimodal AI agents. This page draws only on the paper, the project page, the official GitHub repository and the Hugging Face model card. The model card says Magma is for research purposes and is not intended for production deployment, and every result mentioned here is the authors’ own.

Per the paper, Set-of-Mark (SoM) overlays numbered marks on candidate actionable regions or points in an image, such as clickable buttons in a UI screenshot or a robot arm, and the model predicts which numeric mark to act on. The paper calls this action grounding. Trace-of-Mark (ToM) extends the idea along time: given marks in a frame, the model predicts where they move over the next frames, which the abstract describes as object movements such as the trace of human hands or robotic arms, used for action planning. The paper says ToM pushes the model to look ahead before acting and uses far fewer tokens than next-frame prediction to cover a longer horizon. Per its Table 1, SoM is applied to UI screenshots, robotics images and instructional videos, and ToM to robotics images and instructional videos but not UI data, which is a sequence of discrete screenshots. Coordinates are normalized and quantized into 256 bins. Traces come from the CoTracker point-tracking model. The pipeline drops low-motion traces, applies a homography transform to remove global camera motion, and filters clips whose CLIP similarity to their text is below 0.25.

Per the paper, Magma takes past visual observations and a text task description and outputs tokens that are either verbal or spatial, decoded autoregressively. By default it uses LLaMA-3-8B as the language backbone and ConvNeXt-XXLarge as the vision backbone, with Magma-8B listed at 8.6B parameters in the paper’s results table. All outputs are written as text tokens. 2D UI actions become a textual dictionary, and robot actions use the last 256 discrete language tokens, following OpenVLA. For robot manipulation the paper describes 6-DoF end-effector displacements plus, in some cases, a gripper open or closed dimension; the model card says 7 DoF.

Pretraining runs for at most three epochs at a constant learning rate of 1e-5, tuning the whole model, language model and vision encoder included. The model card adds a batch size of 1024, a maximum sequence length of 4096 and bf16 training on H100 and MI300 GPUs. The authors evaluate zero-shot and after fine-tuning on UI navigation (Mind2Web, AITW), simulated robot benchmarks and a WidowX arm with four tasks of roughly 50 trajectories each. Their ablation reports that simply combining UI and robotics data did not help, and that applying SoM and ToM across the data is what made the mixture work. This page reproduces no scores.

The paper states a training corpus of approximately 39 million diverse samples from four groups. About 2.7M UI screenshots come from SeeClick and Vision2UI. The robotics data is 970K Open X-Embodiment trajectories, 9.4M image-language-action triplets. The video data is over 25M samples from about 4M shot-consistent clips, drawn from Epic-Kitchens, Ego4D, Something-Something v2 and other instructional videos. The multimodal understanding data is 1.2M image-text pairs from ShareGPT4V, LLaVA-1.5 and OCR datasets, which the paper also denotes Magma-SFT (820K). How 1.2M and 820K relate is Not stated in the sources read, and neither is how the 39M figure relates to the roughly 29M images the paper mentions in its spatial-reasoning section.

The paper notes that instructional videos, including egocentric ones, contain significant camera movement, which the homography step removes. The README lists released Open X-Embodiment data with traces, video data with traces, the Magma-820K annotations and training code, and its to-do list still shows the SeeClick and Vision2UI pretraining data with SoM unchecked. Code and the model card are under the MIT licence; dataset licences are Not stated in the sources read. The model card says personal data was filtered or scrubbed from image and text sources, and face anonymization of the videos is Not stated in the sources read.

Per the vision-language-action model entry on this site, a VLA model takes a visual scene and a language instruction and outputs an action. Magma fits that description for robot control, and its model card says it has general agentic capabilities as a vision-language-action model, but its scope is wider: it also answers image and video questions and navigates UIs. Per the OpenVLA entry, OpenVLA has 7 billion parameters, a Llama 2 backbone with a fused DINOv2 and SigLIP encoder, and is fine-tuned solely on 970k Open X-Embodiment robot trajectories.

The Magma paper says it follows OpenVLA in representing robot actions and in using Open X-Embodiment, then adds UI, video and image-text data plus SoM and ToM labels. The authors report Magma outperforming OpenVLA in their zero-shot simulation and fine-tuning comparisons. Those are results on their own tasks, and Firsthand draws no comparison. The first comparison table sets out only what each source or site entry states.

Magma’s training data is the set of public datasets its authors list, and Firsthand is not affiliated with them. Firsthand makes no claim that its data was used with, trained or is compatible with Magma. What the site states instead is custom egocentric collection of real human demonstrations to a written spec, fast and on demand: head and wrist camera video, calibrated stereo depth, 200 Hz IMU, 3D hand pose at 21 joints per hand with a per-joint visibility flag, wrist trajectories, action segments and contact events. The only language field is one task instruction per episode. Magma’s ToM follows the motion of marked points, including human hands, in video; the site states no equivalence between that and its own hand pose or wrist trajectories.

This page makes no claim that Firsthand provides UI or web-agent data, Set-of-Mark or Trace-of-Mark annotations, or video captions, and the site states no mapping from its fields to Magma’s token format or action space. Confirm a spec on a Spec Pilot, a small fast first batch. The median from signed spec to first validated batch is 48 hours where coverage is deep and three to four weeks where it is thin. Billing is per validated hour, and delivery is RLDS, LeRobot, WebDataset or HDF5.

What is Magma?

A foundation model for multimodal AI agents that keeps vision-language understanding and adds planning and acting, from UI navigation to robot manipulation. It is pretrained on images, videos and robotics data labeled with Set-of-Mark and Trace-of-Mark.

What are Set-of-Mark and Trace-of-Mark?

Set-of-Mark overlays numbered marks on actionable objects in an image so the model can ground an action. Trace-of-Mark labels how marked objects move over later video frames, such as the trace of a hand or robot arm, for action planning.

What data was Magma pretrained on?

About 39 million samples, per the paper: UI screenshots, 970K Open X-Embodiment trajectories, over 25M video samples from instructional videos including Ego4D and Epic-Kitchens, and image-text pairs.

Is Magma open source?

The GitHub repository and model card are under the MIT licence. The model card says it is for research only and not intended for production. Dataset licences are Not stated in the sources read. Read the licence text; this is not legal advice.

How does Magma differ from OpenVLA?

OpenVLA is fine-tuned solely on Open X-Embodiment robot trajectories. Magma pretrains on a wider mix of UI, video, robotics and image-text data with Set-of-Mark and Trace-of-Mark labels, and also answers image and video questions.

Does Firsthand use Magma, and can I request a custom dataset?

Firsthand is not affiliated with the Magma authors and makes no claim of use, training or compatibility. You can request a custom egocentric dataset: write a spec, confirm it on a Spec Pilot, pay per validated hour.