What Is the RoboMIND Dataset?

RoboMIND is introduced in RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulation (arXiv 2412.13877) by authors at the Beijing Innovation Center of Humanoid Robotics, Peking University and the Beijing Academy of Artificial Intelligence. The project page and the Hugging Face page say it was accepted by Robotics: Science and Systems (RSS) 2025. The abstract argues that robot datasets need detailed observations and manipulation actions, so they cannot be sourced from the internet like vision or language data, and that existing work lacks a unified data collection standard and enough high-quality data across tasks, scenarios and robot types. RoboMIND answers with one data collection platform and one standardized protocol. The name stands for Multi-embodiment Intelligence Normative Data. The project page also announces RoboMIND V2.0 on ModelScope, which this entry does not describe.

RoboMIND is third-party research. Firsthand is not affiliated with its authors, makes no claim that any named model was trained on Firsthand data, and makes no claim that Firsthand data is part of, compatible with or used alongside RoboMIND. Firsthand captures first-person human video, collected fast, on demand and to a written spec. It is not robot or teleoperation data. The fields the site states are head and wrist camera video, depth derived from a calibrated stereo rig, 3D hand pose at 21 keypoints per hand with a per-joint visibility flag, wrist trajectories, action segments, and 6-DoF IMUs at 200 Hz on one clock. The only language field is one task instruction per episode.

The abstract states 107k demonstration trajectories across 479 tasks involving 96 object classes, with multi-view observations, proprioceptive robot state and linguistic task descriptions. The paper adds 305.5 hours of interaction data and 38 operational skills. It also includes 5k real-world failure demonstrations with their causes, and a digital twin in the Isaac Sim simulator that replicates the real tasks and assets. The paper annotates 10k trajectories with frame-level language descriptions, first generated by Gemini and then manually refined.

The paper sorts tasks into six types: articulated, coordination, basic, multiple object interactions, precision and scene understanding. Objects come from five scenarios: domestic, industrial, kitchen, office and retail. The Hugging Face page lists a version history: version 1.0 had 55K trajectories, 279 tasks and 69 object classes, and versions 1.1 and 1.2 reach the 107K, 479 and 96 figures. The paper names its own limitations as relatively simple backgrounds and no mobile manipulation data.

The paper describes a cloud-native data platform that collects, stores, filters, classifies and annotates the data. Each trajectory is stored as one H5 file holding multi-view RGB-D data, robot proprioceptive state, end-effector state and teleoperation body state. Reviewers check the videos in three steps: an initial inspection, a frame-by-frame detailed inspection, and filtering with issue logging. They check against eight criteria, including unnecessary contact, jerky movement and image distortion.

Teleoperation differs by robot. For single-arm robots the authors follow Gello, with 3D-printed components and servo motors that match the arm’s degrees of freedom. For the dual-arm robot they use a bilateral device similar to Mobile ALOHA. For the humanoid they map an Xsens motion capture suit onto the robot. Each task is named by its embodiment, skill, objects and scene description, and operators work on a rotating rest system to limit fatigue.

The abstract names four embodiments: the Franka Emika Panda, the UR5e, the AgileX dual-arm robot, and the X-Humanoid Tien Kung humanoid with dual dexterous hands. The Hugging Face page gives per-embodiment counts of 52,926 trajectories for Franka, 19,152 for Tien Kung, 10,629 for AgileX and 25,170 for UR5e. The paper’s introduction gives a different breakdown that separates out 30,035 simulation trajectories, so the sources do not agree on one split and each is quoted with its source.

The paper says Franka and UR5e trajectories typically run under 200 time steps, and Tien Kung and AgileX trajectories typically run over 500. Cameras differ by robot: three RealSense D435i cameras on Franka, built-in cameras on Tien Kung and AgileX, and one external top camera on UR5e. Tien Kung uses two Inspire-Robots RH56DFX dexterous hands. The paper notes that most earlier large datasets focus on two-finger grippers and lack dexterous hands.

Each dataset’s own sources set its scale, and the units differ, so the numbers are not like for like. RoboMIND states 107k trajectories, 479 tasks and four embodiments. The site’s Open X-Embodiment entry states 1M+ real robot trajectories from 22 embodiments, pooled from 60 existing datasets and stored as RLDS. The site’s AgiBot World entry states over 1 million trajectories, 2976.4 hours and one humanoid platform. The site’s RH20T entry states over 110,000 contact-rich sequences. The RoboMIND paper has its own comparison table that counts other datasets differently, which is not reproduced here.

The RoboMIND authors describe Open X-Embodiment as compiled from laboratories with differing collection standards and platforms, and RoboMIND as gathered in one standardized setting. That is their view of other work. The structural contrast the sources support is that RoboMIND mixes four embodiments under one protocol, Open X-Embodiment pools many existing datasets, and AgiBot World centres on one humanoid platform.

The site’s teleoperation entry says teleoperation collects demonstrations by a human driving the robot, so data matches the robot’s embodiment exactly but is slow and hardware-bound. RoboMIND sits on that side: proprioceptive and end-effector state with multi-view RGB-D from four named robots, driven through Gello-style devices, a bilateral arm or a motion capture suit. Egocentric human capture records people doing tasks directly, which is faster and more diverse, at the cost of an embodiment gap to close.

Egocentric human video has no robot sensors, so it cannot supply robot state or action. What it can show is how people do a task from the actor’s own viewpoint. The RoboMIND paper’s related work notes that one line of research uses egocentric human videos to assist robot action learning. That is a statement about the field, not about Firsthand data, and RoboMIND itself is robot teleoperation data.

RoboMIND is a dataset of robot trajectories, not a source of first-person human video, and its licence is not stated in the sources read, so check the official pages before use. When a program needs first-person human video of particular tasks, environments or people, write them into a spec and confirm it on a Spec Pilot, a small fast first batch. Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks. Billing is per validated hour, and delivery is in the format your training stack already reads: RLDS, LeRobot, WebDataset or HDF5. Firsthand does not deliver robot or teleoperation data, so any robot demonstrations come from your own setup.

What is the RoboMIND dataset?

A robot manipulation dataset and benchmark: 107k human-teleoperated demonstration trajectories across 479 tasks and 96 object classes on four robot embodiments, collected under one standardized protocol.

What does RoboMIND stand for and which robots does it cover?

Multi-embodiment Intelligence Normative Data for Robot Manipulation. It covers the Franka Emika Panda, the UR5e, the AgileX dual-arm robot and the X-Humanoid Tien Kung humanoid with dual dexterous hands.

How large is RoboMIND?

The abstract states 107k trajectories, 479 tasks and 96 object classes. The paper adds 38 skills and 305.5 hours. The Hugging Face page lists version 1.0 at 55K trajectories and versions 1.1 and 1.2 at 107K. Per-embodiment splits differ between the paper and the Hugging Face page.

What data does each trajectory have?

Per the paper, multi-view RGB-D, proprioceptive robot state, end-effector state and teleoperation body state in one H5 file, plus task language descriptions. Frame-level descriptions cover 10k trajectories, and 5k failure demonstrations come with their causes.

Can RoboMIND be used commercially?

The dataset licence is not stated in the sources read for this entry. The GitHub toolkit repository lists Apache License 2.0, which covers the toolkit, not necessarily the data. Read the official pages yourself; this is not legal advice.

Does Firsthand use RoboMIND, and can I request a custom dataset?

Firsthand is not affiliated with the RoboMIND authors, makes no claim that any model was trained on Firsthand data, and makes no claim that its data is part of, compatible with or used alongside RoboMIND. You can request a custom egocentric dataset: write the tasks, fields and format into a spec, confirm it on a Spec Pilot, and billing is per validated hour. The only language field is one task instruction per episode.