What Is the H2O (Two Hands and Objects) Dataset?

H2O is introduced in H2O: Two Hands Manipulating Objects for First Person Interaction Recognition (arXiv 2104.11181), published at ICCV 2021, with an official project page, an official GitHub repository and an official dataset access page. The authors say it is, to the best of their knowledge, the first benchmark that enables the study of first-person actions using the pose of both left and right hands manipulating objects. They argue that existing first-person interaction datasets mostly provide only 2D features such as bounding boxes and hand segmentation, and that earlier datasets with 3D annotations mainly focus on single-hand manipulation. They also contrast H2O with motion-capture datasets, which they say can only be captured in controlled settings and leave visible markers on the images; H2O annotations are markerless. Beyond the dataset, the paper proposes a method that jointly estimates the 3D pose of two hands and the 6D pose of the manipulated objects from RGB images, and a graph convolutional network that predicts interaction classes. This page describes the dataset and does not repeat the paper’s accuracy results.

H2O is third-party research. Firsthand is not affiliated with its authors or their institutions, and makes no claim that H2O uses Firsthand data or hardware or that any model was trained on Firsthand data. The site states custom egocentric collection to a written spec, fast and on demand, delivering first-person human video with 3D hand pose at 21 joints per hand and a per-joint visibility flag, calibrated stereo depth and 6-DoF IMUs at 200 Hz. The only language data tied to Firsthand capture is one task instruction per episode. Firsthand captures first-person human video, not robot or teleoperation data.

Per the paper, H2O was acquired in indoor settings with five Azure Kinect cameras giving synchronized RGB and depth images: four static cameras placed at arbitrary locations that cover the hand-object interactions, and one egocentric camera mounted on the forehead of a helmet worn by the participants. The paper reports 1280x720 resolution for both RGB and depth at 30 fps. The cameras are synchronized with physical cables, and the paper reports a total maximum delay between cameras of 0.74 ms. Extrinsic calibration uses nine IR reflective spheres and a PnP solution, while the intrinsics are the factory-calibrated values from the Azure Kinect.

The paper reports 571,645 RGB-D frames, with four participants performing 36 distinct action classes with eight objects in three environments: a hall, an office and a kitchen, using several backgrounds. The split is subject-based. One subject is left out for testing, and the paper reports 344,645 frames for training, 73,380 for validation and 153,620 for testing. It also reports that both hands are used in 57.8% of the dataset, only the left hand in 12.4% and only the right hand in 29.8%. Total recording duration in hours is Not stated in the sources read.

The abstract lists synchronized multi-view RGB-D images, interaction labels, object classes, ground-truth 3D poses for left and right hands, 6D object poses, ground-truth camera poses, object meshes and scene point clouds. The annotations come from a semi-automated, markerless pipeline. Each object is scanned with a Kinect to build a mesh, object pose is tracked with DenseFusion and refined with ICP, and hands are tracked by fitting the MANO parametric hand model to multi-view depth data, which gives 21 joint locations per hand in the paper. The authors manually detect failure cases, remove those poses and replace them through temporal smoothing.

Action labels are verb-noun pairs selected manually over the sequences with the VIA annotation tool: 11 verb classes, 8 noun classes and, after excluding pairs that do not occur, 36 action classes, with one verb and one noun per frame. To check the poses, the paper annotates 500 images on 5 camera views by hand and says the error for both hands and the object is approximately within a range of 1 cm. The GitHub README documents per-camera folders for RGB, depth, camera pose, hand pose, object pose and action labels, with a hand-pose record of 21 3D points per hand plus an annotated flag. Language beyond these verb-noun labels, such as free-form instructions or captions, is Not stated in the sources read. Inertial data is Not stated in the sources read.

The paper frames H2O around egocentric interaction recognition and joint hand-object pose estimation, and it trains and evaluates baselines for both on the subject-based split. It also says the dataset facilitates modelling hand-object contact, illustrating this with contact maps computed on the hand and object meshes from nearest-vertex distances within a threshold. The README links the dataset to two challenges, one on hand pose and one on action, tied to an ECCV 2022 workshop on egocentric and multi-view cameras.

The paper’s conclusion names robotics among the communities it believes may be interested. Training robot policies on H2O is Not stated in the sources read, and this page draws no conclusion about how H2O would perform as pretraining data for any robot model. H2O records human hands only. For the site’s framing of egocentric human data for robot imitation, see the EgoMimic entry.

H2O and custom egocentric capture answer different questions. H2O is a laboratory research dataset: four participants, a fixed set of eight objects, three environments, and a rig of five RGB-D cameras with one head-mounted view. Custom egocentric capture is a data service: paid, consented contributors recording first-person video to a written spec, with fields and format set in advance. This page makes no ranking between them.

The comparison table below sets only what the official sources state against what the site states, and where a source is silent the cell says so. One difference matters for planning. The terms of use on the official access page say the dataset is to be used only for academic purposes, will not be used for commercial purposes and will not be transferred to any third party. Licence terms in the GitHub README are Not stated in the sources read, so a team with a commercial program should read the official terms directly rather than rely on this summary.

When a program needs tasks, objects, environments or people that an existing dataset does not cover, specified in writing, and the data has to be usable for the program’s own purpose. For two-handed work the site states that both hands stay in frame with per-hand 3D pose and visibility flags, including long continuous clips of fine bimanual work. If contact signals matter, ask for the separate pilot-ready tactile glove tier, time-synced by hardware timestamps and an IMU.

Confirm the spec on a Spec Pilot, a small fast first batch. Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks. Billing is per validated hour, and delivery is in the format your training stack already reads: RLDS, LeRobot, WebDataset or HDF5.

What is the H2O dataset?

A published egocentric dataset of two hands manipulating objects, with 571,645 synchronized RGB-D frames, 3D hand poses, 6D object poses, camera poses and action labels, from an ICCV 2021 paper.

What does H2O stand for?

The paper expands the name as 2 Hands and Objects.

How many views does H2O have?

Per the paper, five Azure Kinect cameras: four static cameras plus one egocentric camera on a helmet, recording RGB and depth at 1280x720 and 30 fps.

What annotations does H2O include?

Per the abstract: interaction labels, object classes, ground-truth 3D poses for left and right hands, 6D object poses, camera poses, object meshes and scene point clouds. Action labels are verb-noun pairs.

Can H2O be used commercially?

The terms of use on the official access page say the dataset is to be used only for academic purposes and will not be used for commercial purposes. Licence terms in the GitHub README are Not stated in the sources read.

Does H2O include language annotations?

Only verb-noun action labels are described. Free-form language such as instructions or captions is Not stated in the sources read.

Does Firsthand use H2O, or is it affiliated with its authors?

No. Firsthand is not affiliated with the H2O authors or their institutions, and makes no claim that H2O uses Firsthand data or hardware or that any model was trained on Firsthand data.

Can I request a custom egocentric dataset?

Yes. Write the tasks, environments, fields and format into a spec, confirm it on a Spec Pilot, and billing is per validated hour. Firsthand captures first-person human video, not robot or teleoperation data.