What Is the RH20T Dataset?

RH20T is introduced in RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot (arXiv 2307.00595) by Hao-Shu Fang, Hongjie Fang, Zhenyu Tang, Jirong Liu, Chenxi Wang, Junbo Wang, Haoyi Zhu and Cewu Lu of Shanghai Jiao Tong University, with an official project page and a GitHub repository for its data API. The abstract says one-shot imitation learning has focused on simple cases such as push or pick-place tasks that rely on visual guidance alone, while many complex skills may need visual and tactile perception. The authors aim to let an agent generalize to hundreds of real-world skills with multi-modal perception. The arXiv abstract also lists a language description for each robot sequence; the project page abstract does not repeat it, and how descriptions are written is Not stated in the sources read. No official Hugging Face page was found, so none is linked. The full paper body was not read for this entry.

RH20T is third-party research. Firsthand is not affiliated with its authors, makes no claim that any named model was trained on Firsthand data, and makes no claim that Firsthand data is part of, compatible with or used alongside RH20T. Firsthand captures first-person human video, collected fast, on demand and to a written spec. It is not robot or teleoperation data, and standard capture lists no tactile or force stream. The fields the site states are head and wrist camera video, depth derived from a calibrated stereo rig, 3D hand pose at 21 keypoints per hand with a per-joint visibility flag, wrist trajectories, action segments, and 6-DoF IMUs at 200 Hz on one clock. The only language field is one task instruction per episode.

The project page lists the streams in a table: RGB images at 1280 x 720 and 10 Hz, depth images, binocular infrared images, robot joint angle and joint torque at 10 Hz, gripper Cartesian pose at 100 Hz, gripper width, 6-DoF force and torque at 100 Hz, audio, and fingertip tactile sensing at 200 Hz. The tactile stream is available only in robot configuration 7, and the sizes of the joint and pose arrays depend on the robot type.

The page says all cameras are calibrated to the robot base frame and the recorded data are synchronized in time. Each scene’s metadata carries a task completion rating, where 0 means robot failure, 1 means task failure and 2 to 9 give completion quality, plus a calibration quality value. How many scenes, users and hours the full dataset holds is Not stated in the sources read.

The project page says the tasks are 48 from RLBench, 29 from MetaWorld and 70 self-proposed tasks that robots often meet and can achieve. Operators teleoperate with a haptic device and a pedal. The page contrasts this with earlier setups that use 3D mice, VR remotes or mobile phones, and stresses intuitive, accurate teleoperation for contact-rich data.

Each platform has a robot arm with a force-torque sensor, a gripper, one or two in-hand cameras, eight to ten global RGBD cameras and two microphones, all linked to a data collection workstation. The download listing names seven robot configurations, and the page points to the paper appendix for their details, which was not read here. The page warns that the data are volunteer-recorded human-robot interactions that may show volunteers’ faces and voices, and asks that it be used solely for model training.

Per the abstract, recent one-shot imitation learning research shows promise in transferring trained policies to new tasks based on demonstrations, which the authors call attractive for acquiring new skills and for improving task and motion planning. They argue that limits in training data have kept the community on simple cases that use visual guidance alone.

RH20T pairs each robot sequence with a corresponding human demonstration video, and its data format stores a human folder beside each robot scene. Whether the authors train or evaluate a one-shot policy on it, which methods they test and any success rates are Not stated in the sources read, so none are reproduced here.

Each dataset’s own sources set its scale, and the units differ, so the numbers are not like for like. RH20T states over 110,000 contact-rich sequences across diverse robots and camera viewpoints. The site’s Open X-Embodiment entry states 1M+ real robot trajectories from 22 embodiments, pooled from 60 existing datasets and stored as RLDS. The site’s DROID entry states 76k successful teleoperated trajectories, 350 hours, 564 scenes and 86 tasks on one shared hardware stack.

Of the three, only RH20T is described in the sources read with force, audio and a paired human demonstration video for each robot sequence; the site’s entries for the other two do not describe them. Elsewhere on this site, the SpatialVLA entry says its README and Hugging Face card name Open X-Embodiment and RH20T as sources of demonstrations, and the GR00T N1 comparisons list RH20T-Human among seven existing human video datasets. Those are statements about other papers, not about Firsthand.

The site’s teleoperation entry says teleoperation collects demonstrations by a human driving the robot, so data matches the robot’s embodiment exactly but is slow and hardware-bound. Egocentric human capture records people doing tasks directly, which is faster and more diverse, at the cost of an embodiment gap to close. RH20T sits on the teleoperation side: its streams include joint angles, joint torque, gripper pose, and force and torque read from the robot itself.

Egocentric human video has no robot sensors, so it cannot supply those signals. What it can show is how people do a task. RH20T’s own human demonstration videos are a different thing from first-person capture: the camera viewpoint of those videos is Not stated in the sources read, so this page does not call them egocentric.

RH20T is a dataset of robot sequences, not a source of first-person human video, and its data and code come from its authors. When a program needs first-person human video of particular tasks, environments or people, write them into a spec and confirm it on a Spec Pilot, a small fast first batch. Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks. Billing is per validated hour, and delivery is in the format your training stack already reads: RLDS, LeRobot, WebDataset or HDF5. Firsthand does not deliver robot or teleoperation data, so any robot demonstrations come from your own setup.

What is the RH20T dataset?

A robot manipulation dataset from a 2023 paper: over 110,000 contact-rich, real-world sequences, each with visual, force, audio and action information plus a human demonstration video. It is built to study one-shot skill learning.

What does each RH20T sequence contain?

Per the abstract, visual, force, audio and action information, plus a corresponding human demonstration video and a language description for each robot sequence. The project page lists RGB, depth, binocular infrared, joint, gripper, force-torque and audio streams, with fingertip tactile in robot configuration 7 only.

How large is RH20T?

The project page says the original size is 40TB, so it provides a 640 x 360 version of about 5TB for RGB and about 10TB for RGBD, and a 320 x 180 version. The number of scenes and hours is Not stated in the sources read.

Can RH20T be used commercially?

The project page says the dataset is licensed under a mixture of licences and is divided into RH20T-C (commercial) and RH20T-NC (non-commercial). The GitHub API repository lists the MIT License. Read each licence yourself; this is not legal advice.

Does RH20T include human video?

Per the abstract, each robot sequence has a corresponding human demonstration video. The project page warns the data may show volunteers’ faces and voices. The camera viewpoint of those videos is Not stated in the sources read.

Does Firsthand use RH20T, and can I request a custom dataset?

Firsthand is not affiliated with the RH20T authors, makes no claim that any model was trained on Firsthand data, and makes no claim that its data is part of, compatible with or used alongside RH20T. You can request a custom egocentric dataset: write the tasks, fields and format into a spec, confirm it on a Spec Pilot, and billing is per validated hour. The only language field is one task instruction per episode.