What Is HOI4D?

HOI4D is introduced in HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction (arXiv 2203.01577, CVPR 2022), with an official project page and an official GitHub instructions repository. The authors say most earlier datasets covered instance-level interaction, where objects come from a very small pool with known CAD models, and that HOI4D instead targets category-level variation across both rigid and articulated objects.

HOI4D is third-party research. Firsthand is not affiliated with its authors or their institutions, and makes no claim that HOI4D uses Firsthand data or hardware or that any model was trained on Firsthand data. The site states custom egocentric collection to a written spec, fast and on demand, with 3D hand pose at 21 joints per hand, calibrated stereo depth and 6-DoF IMUs at 200 Hz.

Per the paper, participants wore a bicycle helmet carrying a Kinect v2 and an Intel RealSense D455 RGB-D sensor, pre-calibrated and synchronized before capture. The authors describe the Kinect v2 as time-of-flight and suited to longer range, and the RealSense D455 as structured light with advantages within about one metre. Each video was captured at 15 fps for 20 seconds.

Participants carried out tasks without wearing markers, in indoor rooms. Tasks were split into a simple level, a pick-and-place with a relatively clean background, and a complex level, with objects placed in a cluttered manner. The paper’s abstract gives 4 participants while its introduction and the project page give 9, so this page relies on neither count.

Frame-wise annotations cover panoptic segmentation, motion segmentation, 3D hand pose, category-level object pose and hand action, together with reconstructed object meshes and scene point clouds. Hand pose uses the MANO parametric hand model on a 21-keypoint hand definition, and the README documents MANO pose and shape parameters plus 21 keypoint projections per video.

Annotation mixed human work with automatic algorithms: manual labels on sampled frames (hand keypoints, oriented object boxes, motion masks) were propagated or optimized across each sequence, with failures corrected by hand. Per the README, only part of the data is on the released list and the rest is temporarily kept as a test set.

The authors set three benchmarking tasks: semantic segmentation of 4D dynamic point cloud sequences, category-level object pose tracking, and egocentric action segmentation with diverse interaction targets. They report that HOI4D poses great challenges to existing methods. Those are the authors’ results on their own baselines, and Firsthand quotes no numbers from them.

The paper adds that categories were mainly selected from ShapeNet and Sapien Assets to ease sim-to-real study, and that the meshes and hand trajectories could potentially be put into simulation. Its stated main limitation is that two-handed manipulation tasks are not covered. The README points to a HOI4D Challenge on a newer project website.

HOI4D is a fixed public release built for category-level research: many object instances per category, a helmet-mounted two-sensor RGB-D rig, markerless capture in indoor rooms, and labels from annotation pipelines plus optimization. Custom egocentric capture is a different design: paid, consented contributors working to a written spec, with 3D hand pose at 21 joints per hand, calibrated stereo depth and IMUs.

The site does not state object pose as a delivered field, so object pose is scoped per spec and confirmed on a Spec Pilot. This page makes no accuracy comparison between the two.

When a program needs tasks, environments or objects that a published dataset does not cover, specified in writing. HOI4D’s own paper lists two-handed manipulation as not covered, and whether a given spec fits is confirmed on a Spec Pilot. Firsthand does not build or train models. If contact signals matter, ask for the separate pilot-ready tactile glove tier.

A Spec Pilot is a small fast first batch. Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks. Billing is per validated hour.

What is HOI4D?

A released 4D egocentric dataset for category-level human-object interaction, with RGB-D video and frame-wise hand pose, object pose, segmentation and action annotations.

What does HOI4D contain?

2.4M RGB-D egocentric frames over 4000 sequences, with 800 object instances from 16 categories in 610 indoor rooms, plus reconstructed object meshes and scene point clouds.

What does category-level mean in HOI4D?

Objects come in many instances per category rather than a small pool of known CAD models, so methods can be studied on potentially unseen objects, including articulated ones.

Is HOI4D open?

The official instructions repository is under the MIT License. The dataset licence is Not stated in the sources read, and part of the data is kept as a test set.

Does HOI4D cover two-handed tasks?

No. The paper lists two-handed manipulation as the main limitation and not covered.

Does Firsthand use HOI4D?

No. Firsthand is not part of it or affiliated with its authors or their institutions, and makes no claim that HOI4D uses Firsthand data or hardware or that any model was trained on Firsthand data.

Can Firsthand collect hand-object data to a custom spec?

The site states custom egocentric collection to a written spec, fast and on demand, with 21-joint 3D hand pose, calibrated stereo depth and 200 Hz IMUs, confirmed on a Spec Pilot.