What Is RoboBrain?

RoboBrain is introduced in the paper RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concrete (arXiv 2502.21257) by Yuheng Ji, Huajie Tan, Jiayu Shi, Xiaoshuai Hao and co-authors at the Beijing Academy of Artificial Intelligence and partner institutions. The authors argue that general multimodal large language models fall short on long-horizon manipulation, and pair the model with a new dataset, ShareRobot, built to teach planning, affordance perception and trajectory prediction.

RoboBrain is third-party research. Firsthand is not part of it or affiliated with its authors, and makes no claim that any model was trained on Firsthand data. What the site states is custom egocentric collection of real human demonstrations, fast and on demand, to a written spec: 3D hand pose at 21 joints per hand, calibrated stereo depth and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset with a buyer-owned commercial license. The only language data the site states is one task instruction per episode. Firsthand captures hand-object signals, not affordance labels.

The paper says current multimodal language models lack three capabilities that long-horizon manipulation needs: planning, which decomposes a complex instruction into sub-tasks; affordance perception, which recognizes how interactive objects can be used; and trajectory prediction, which anticipates the full path of a manipulation.

The authors illustrate this with a robotic arm lifting a teapot and pouring water into a cup. The model has to split the task into sub-tasks, find the graspable region of the teapot and predict the path to it. They attribute the gap mainly to a scarcity of large-scale, fine-grained datasets designed for robotic operation.

In the paper, abstract means a high-level instruction and concrete means an explicit output the robot can act on. In practice the model first generates a detailed plan, splits it into sub-task descriptions, and then predicts affordance and trajectory for each sub-task.

The paper defines affordance as the area where a human hand makes contact with an object, written as a bounding box with top-left and bottom-right corners. It defines a trajectory as a 2D visual trace: a series of 2D coordinates for the end-effector or hand over the episode.

RoboBrain has three modules: a foundational model for planning, an A-LoRA module for affordance and a T-LoRA module for trajectory. The foundational model follows the LLaVA design, with a SigLIP vision encoder, a 2-layer MLP projector and Qwen2.5-7B-Instruct as the language model.

ShareRobot is a dataset the authors built to label task planning, object affordance and end-effector trajectory. It starts from robot demonstrations in the Open X-Embodiment collection: 51,403 instances selected from 23 original datasets, covering 102 scenes and 12 robot embodiments.

The selection rules keep successful demonstrations with clear descriptions, visible affordances and clear trajectories, and drop low-resolution, failed or occluded ones. Planning labels are generated from 30 frames per demonstration and refined by three human annotators, producing 1,027,990 question-answer pairs. Affordance labels cover 6,522 images and trajectory labels cover 6,870 images.

Training has two phases. Phase 1 is general multimodal training, in the style of LLaVA-OneVision, to build a base model with broad understanding and instruction following. Phase 2 is robotic training. Stage 3 tunes the full model on about 1.3M robotic samples drawn from RoboVQA, scene-scanning datasets and a subset of ShareRobot. Stage 4 adds LoRA modules for affordance and trajectory.

The authors evaluate planning on RoboVQA, OpenEQA and the ShareRobot test set, affordance on the AGD20K test set, and trajectory on ShareRobot with several model variants. They report state-of-the-art performance on the planning benchmarks and a BLEU-4 score on RoboVQA that exceeds the second-place model by 18.75.

These are the authors’ claims on their own setups, and Firsthand does not reproduce any of them.

A vision-language-action model, as the site defines it, conditions on images and a language instruction and outputs actions. The paper describes RoboBrain as outputting plans, affordance regions and 2D trajectory waypoints. Motor-action output is Not described in the paper text read, so the entry does not classify it as a vision-language-action model.

It sits next to the vision-language model, which the site defines as a model that does not output an action by itself, and the robot foundation model, a large policy pretrained on broad robot data. Firsthand ranks none of these models.

This entry draws on the arXiv paper, the official GitHub repository and the project page. The repository page lists the Apache License 2.0 and links checkpoints and the dataset. Whether each checkpoint carries a separate licence is Not stated here.

The repository also lists RoboBrain 2.0, and the project page now presents RoboBrain 2.5. This entry covers the original paper only. ShareRobot is built from robot demonstrations, and the paper does not state that human egocentric video, stereo depth, hand pose or IMU data was used, so how those would suit RoboBrain is Not stated.

When a program needs real human egocentric demonstrations of tasks, environments or objects that public data does not cover, specified in writing. Firsthand does not build or train models, and the language it delivers is one task instruction per episode. Planning, affordance and trajectory labels are not part of what the site states.

Confirm the spec on a Spec Pilot, a small fast first batch. Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks. Billing is per validated hour.

What is RoboBrain?

A published multimodal large language model for robotic manipulation (arXiv 2502.21257, CVPR 2025) that plans a task in steps, then predicts object affordance regions and 2D end-effector trajectories.

What does RoboBrain predict?

Per the paper, a step-by-step plan, an affordance region as a bounding box for hand contact, and trajectory waypoints as 2D coordinates for the end-effector or hand.

What is ShareRobot?

A dataset built by the RoboBrain authors that labels task planning, affordance and trajectory, with 1,027,990 planning question-answer pairs drawn from 51,403 Open X-Embodiment instances.

Is the code open?

The official GitHub repository lists the Apache License 2.0 and links checkpoints and the dataset. A separate licence for each checkpoint is Not stated in the sources read.

Does Firsthand data train RoboBrain?

No claim is made. Firsthand is not affiliated with its authors, and the site does not state that any model was trained on Firsthand data.

Does Firsthand deliver affordance or trajectory labels?

The site states hand-object signals, not affordance labels, and one task instruction per episode. Planning and trajectory labels are Not stated. Scope is confirmed on a Spec Pilot.