What Is AgiBot World?

AgiBot World is introduced in AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systems (arXiv 2503.06669) by Team AgiBot-World, a joint effort of The University of Hong Kong, AgiBot Inc., Shanghai Innovation Institute and Shanghai AI Lab, with an official project site, a GitHub repository and a Hugging Face dataset page. The abstract describes a platform of over 1 million trajectories across 217 tasks in five deployment scenarios, gathered with a standardized collection pipeline and human-in-the-loop verification, and extensible from grippers to dexterous hands and visuo-tactile sensors. The authors argue that robot datasets suffer from fragmented clips, heterogeneous hardware and unstandardized collection, and that earlier sets lean on short-horizon tasks in controlled labs. Their answer is real-world data from a purpose-built facility on one humanoid hardware platform. The paper also introduces a generalist policy, GO-1.

AgiBot World is third-party research. Firsthand is not affiliated with its authors, makes no claim that any named model was trained on Firsthand data, and makes no claim that Firsthand data is part of, compatible with or used alongside AgiBot World. Firsthand captures first-person human video, collected fast, on demand and to a written spec. It is not robot or teleoperation data. The fields the site states are head and wrist camera video, depth derived from a calibrated stereo rig, 3D hand pose at 21 keypoints per hand with a per-joint visibility flag, wrist trajectories, action segments, and 6-DoF IMUs at 200 Hz on one clock. The only language field is one task instruction per episode.

Per the paper, the latest version holds 1,001,552 trajectories with a total duration of 2976.4 hours, covering 217 specific tasks, 87 skills and 106 scenes. The GitHub README gives the complete Beta release as 1,003,672 trajectories (about 43.8T) and lists an Alpha subset of 92,214 trajectories (about 8.5T). The Hugging Face page says 200+ types of tasks and 100+ real-world scenarios. Each is quoted with its source.

The paper says each episode has multiple camera views, depth information, camera calibration, and language annotations for the overall task and for each sub-step. The Hugging Face page describes a per-task JSON file whose action slices carry start and end frames, an atomic skill and an instruction, and a per-episode HDF5 file of proprioceptive state and action. The paper states that trajectories typically span about 30 seconds, with some over 2 minutes, and that about one percent of the dataset is failure recovery data annotated with reasons and timestamps.

The paper describes a facility of over 4,000 square meters with over 3,000 distinct objects, laid out for five domains: domestic, retail, industrial, restaurant and office. The robots are more than 100 homogeneous units of the authors’ AgiBot G1 platform, with dual 7-DoF arms, a mobile chassis and an adjustable waist. The end effector is a standard gripper, a 6-DoF dexterous hand, or a gripper with visuo-tactile sensors. Eight cameras record, and images and proprioceptive states are logged at 30 Hz.

Skilled teleoperators drive the robots with a VR headset or a whole-body motion capture system. The paper describes three phases: a feasibility pass that sets collection standards, collection with local validity checks such as missing frames, and annotator review with language annotations. The authors also train a policy on early demonstrations and use how it behaves on deployment to revise the protocol. They say this led them to add a step that removes idle frames.

The paper and README use AgiBot World Colosseo for the full platform and AgiBot World for the dataset. The README calls Colosseo a full-stack robot learning platform for bimanual manipulation, accompanied by foundation models, benchmarks and an ecosystem, and lists released items: the GO-1 and GO-1 Air models, a task catalog, and the Beta and Alpha datasets. The README also says all the data and code in the repository are under CC BY-NC-SA 4.0.

The abstract reports that policies pre-trained on the dataset show an average 30% improvement over those trained on Open X-Embodiment, and that GO-1 exceeds a 60% success rate on complex tasks. These are the authors’ own real-world evaluations, and this page neither checks nor extends them. The project site also presents a separate AGIBOT WORLD 2026 dataset on a G2 platform; this entry covers the 2025 Colosseo data.

Each dataset’s own sources set its scale, and the units differ, so the numbers are not like for like. AgiBot World states over 1 million trajectories and 2976.4 hours. The site’s Open X-Embodiment entry states 1M+ real robot trajectories from 22 embodiments, pooled from 60 existing datasets and stored as RLDS. The site’s DROID entry states 76k successful teleoperated trajectories, 350 hours, 564 scenes and 86 tasks on one shared hardware stack. The site’s RH20T entry states over 110,000 contact-rich sequences. The AgiBot paper has its own comparison table with its own counting, which is not reproduced here.

The paper describes Open X-Embodiment as an aggregation with varied embodiments and inconsistent quality, and DROID as crowd-sourced on fixed single-arm hardware without human-feedback quality assurance. Those are the AgiBot authors’ views of other work. The structural contrast the sources support is that AgiBot World is one dual-arm humanoid platform with human-in-the-loop verification, Open X-Embodiment pools many embodiments, and DROID uses one single-arm stack.

The site’s teleoperation entry says teleoperation collects demonstrations by a human driving the robot, so data matches the robot’s embodiment exactly but is slow and hardware-bound. AgiBot World sits on that side: joint, end-effector, head, waist and chassis state and action in a fixed robot frame, from operators using a VR controller or motion capture. Egocentric human capture records people doing tasks directly, which is faster and more diverse, at the cost of an embodiment gap to close.

Egocentric human video has no robot sensors, so it cannot supply robot state or action. What it can show is how people do a task from the actor’s own viewpoint. The AgiBot paper itself draws on human video for its latent action stage, and its figure caption names Ego4D. That is a statement about the paper’s method, not about Firsthand data.

AgiBot World is a dataset of robot trajectories, not a source of first-person human video, and its data and code come from its authors under a non-commercial licence. When a program needs first-person human video of particular tasks, environments or people, write them into a spec and confirm it on a Spec Pilot, a small fast first batch. Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks. Billing is per validated hour, and delivery is in the format your training stack already reads: RLDS, LeRobot, WebDataset or HDF5. Firsthand does not deliver robot or teleoperation data, so any robot demonstrations come from your own setup.

What is AgiBot World?

A large-scale robot manipulation platform and dataset from Team AgiBot-World: over 1 million teleoperated trajectories from more than 100 humanoid robots across five real-world domains, with human-in-the-loop verification and language annotations.

What is AgiBot World Colosseo?

The name of the full platform in the paper and README. The README describes a full-stack robot learning platform with foundation models, benchmarks and an ecosystem, including the GO-1 models, a task catalog and the Beta and Alpha datasets.

How large is the AgiBot World dataset?

The paper states 1,001,552 trajectories, 2976.4 hours, 217 tasks, 87 skills and 106 scenes. The GitHub README lists Beta at 1,003,672 trajectories and Alpha at 92,214. The sources differ slightly, so each is quoted with its source.

What sensors and data does each episode have?

Per the paper, multiple camera views, depth, camera calibration and language annotations for the task and each sub-step, logged at 30 Hz from a robot with eight cameras. Proprioceptive state and action are stored in an HDF5 file per episode.

Can AgiBot World be used commercially?

The paper, README and Hugging Face page all state CC BY-NC-SA 4.0 for the data and code, which includes a non-commercial term. Read the licence yourself; this is not legal advice.

Does Firsthand use AgiBot World, and can I request a custom dataset?

Firsthand is not affiliated with the AgiBot World authors, makes no claim that any model was trained on Firsthand data, and makes no claim that its data is part of, compatible with or used alongside AgiBot World. You can request a custom egocentric dataset: write the tasks, fields and format into a spec, confirm it on a Spec Pilot, and billing is per validated hour. The only language field is one task instruction per episode.