What Is RoboMamba?

RoboMamba is introduced in RoboMamba: Efficient Vision-Language-Action Model for Robotic Reasoning and Manipulation (arXiv 2406.04339), first submitted on 6 June 2024 and last revised on 14 December 2024 by Jiaming Liu and colleagues. The arXiv listing notes acceptance at NeurIPS 2024, and the project page lists Peking University, AI2Robotics and the Beijing Academy of Artificial Intelligence (BAAI). The abstract names two problems with existing VLA models: insufficient reasoning ability on complex tasks, and high computational cost for fine-tuning and inference. RoboMamba is the authors’ answer, an end-to-end robotic VLA that uses Mamba for both reasoning and action. This page draws only on the paper, the project page, the GitHub repository and the Hugging Face paper page; no Hugging Face model card was read.

The paper says robots must respond quickly to human instructions, so efficiency matters. It chooses Mamba for its context-aware reasoning and linear computational complexity, and contrasts this with attention-based language models, which it says make fine-tuning and action generation costlier. Mamba’s selective scan makes the state space parameters depend on the input, which the paper credits with better content-aware reasoning. The authors call their work the first attempt to bring Mamba to robotics. They also skip a vision encoder ensemble, arguing it adds computational cost that hurts real-world practicality. In an ablation with the same data and strategy, the paper reports significant gains for Mamba-2.7B over RWKV-3B, another linear-complexity model.

Images pass through a CLIP visual encoder, and a multilayer perceptron connector maps the features into the language embedding space. Text is tokenized, concatenated with the visual tokens and fed to Mamba. Stage 1.1, alignment pretraining, freezes the encoder and Mamba and updates only the projection layer on the filtered LLaVA 558K image-text set. Stage 1.2, instruction co-training, fine-tunes the projection layer and Mamba on general instruction data plus robotic data from RoboVQA, with cross-entropy loss. The main experiments use the LLaVA 1.5 mixture and 300K RoboVQA samples.

Stage 2 freezes every RoboMamba parameter and adds two MLP policy heads for end-effector position and direction. The pose is 6-DoF for a Franka Emika Panda arm, with gripper state added for grasping. The heads are about 0.1% of the model, 3.7M parameters or 7MB, and need a few dozen minutes on a single A100 GPU. A policy head ablation found little accuracy difference between designs, which the paper reads as showing that reasoning ability drives manipulation learning.

Stage 1 uses web image-text and instruction data plus RoboVQA, which the paper describes as data collected by performing user requests with robots, humans and humans with grasping tools, with 5,246 long-horizon and 92,948 medium-horizon episodes. Stage 2 uses a dataset the authors generate in the SAPIEN simulator, where a Franka Panda with a suction gripper acts on articulated PartNet-Mobility objects: 10K images across 20 tasks for training, and a 1.1K-example test set that adds 10 unseen tasks. The real-world demonstrations use a Franka Emika arm fitted with a tape-based suction gripper on household objects.

Reasoning is scored on VQAv2, OKVQA, GQA, VizWiz, POPE, MME, MMBench, MM-Vet and RoboVQA, including the 18k RoboVQA validation set. The paper notes that some scores trail LLaVA 1.5 and SPHINX and says it prioritized a smaller, faster model. Manipulation is tested open loop in SAPIEN, and a sample counts as a success when the object’s joint state changes by more than 0.1 meters.

The paper reports gains over ManipLLM of 7.0% on seen tasks and 2.0% on unseen tasks, with the best result on 14 of 20 seen tasks, and inference 3 times faster than existing VLA models. It states limits too: the 2.7B language model trails 7B and 13B models on some complex reasoning, and real-world failures include misreading the task objective or target object.

The GitHub repository lmzpai/roboMamba lists an MIT License, gives install and test commands, and puts checkpoints on a test branch. Its README cites the paper under a slightly different title, but the paper and project page text read do not link to it, so treat it as the repository named for the paper. The README says training code is available on request. Licence terms for the paper and for checkpoints beyond the MIT label are Not stated in the sources read. This is not legal advice.

RoboMamba is a model design and training recipe. Firsthand supplies data. RoboMamba’s reported training mixes image-text instruction data, RoboVQA and a simulated set of Franka Panda end-effector poses. Firsthand captures real human egocentric demonstrations from a head-mounted camera and two wrist cameras. Nothing on this site describes robot action or joint data, end-effector pose labels, teleoperation data, simulation data, visual question answering data or video captions, and none is claimed here. The only language field the site states is one task instruction per episode.

Firsthand makes no claim that its data trained, was used with or is compatible with RoboMamba, RoboVQA, SAPIEN or any other model, robot, simulator or benchmark. Firsthand data is real capture, never synthetic, generated or simulated. The site’s general view is in its guide on training data for VLA models, linked below.

Firsthand collects real egocentric demonstrations on demand, to a written spec. The rig is a head-mounted global-shutter camera plus two wrist cameras, with active-stereo depth and 3x IMU on one hardware-triggered PTP clock. Episodes carry 3D hand pose at 21 joints per hand with a per-joint visibility flag, wrist trajectories, contact events, frame-accurate action segments and one task instruction per episode.

You write the spec, confirm it on a Spec Pilot, and pay per validated hour. The site states a 48-hour median turnaround where coverage is deep and three to four weeks where it is thin. Delivery is RLDS, LeRobot, WebDataset or HDF5, with a buyer-owned commercial license. Request a custom dataset below.

What is RoboMamba?

An end-to-end robotic vision-language-action model that pairs a vision encoder with the Mamba state space language model for robotic reasoning and manipulation. It is introduced in arXiv 2406.04339.

Why does RoboMamba use Mamba?

The paper chooses Mamba for its context-aware reasoning and linear computational complexity, aiming to keep fine-tuning and inference efficient compared with attention-based language models.

How does RoboMamba learn manipulation?

After reasoning training, all parameters are frozen and a simple policy head of two MLPs is trained to predict end-effector position and direction. The paper states the head is about 0.1% of the model and takes a few dozen minutes to fine-tune.

How fast is RoboMamba?

The abstract reports inference 3 times faster than existing VLA models. The paper also reports 7 times faster than LLaMA-AdapterV2 and ManipLLM. An absolute control frequency is Not stated in the sources read.

Can RoboMamba be used commercially?

The GitHub repository lists an MIT License. Licence terms for the paper and for checkpoints beyond that label are Not stated in the sources read. Check the repository directly; this is not legal advice.

Does Firsthand use RoboMamba, and can I request a custom dataset?

Firsthand is not affiliated with RoboMamba’s authors and makes no claim of use, training or compatibility. Firsthand data is real human egocentric capture. You can request a custom egocentric dataset: write a spec, confirm it on a Spec Pilot, pay per validated hour.