What Is HOT3D?

HOT3D is introduced in HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos (arXiv 2411.19167, CVPR 2025), with an official project page and GitHub toolkit. The authors say it is primarily intended for training and evaluating methods that track hands and objects in 3D from localized, egocentric, multi-view video, as opposed to single views or individual images.

HOT3D is third-party research. Firsthand is not affiliated with its authors or their institution, and makes no claim that HOT3D uses Firsthand data or hardware or that any model was trained on Firsthand data. The site states custom egocentric collection to a written spec, fast and on demand, with 3D hand pose at 21 joints per hand, calibrated stereo depth and 6-DoF IMUs at 200 Hz.

Per the paper, recordings come from two head-mounted headsets at 30 fps, with image streams synchronized by a hardware trigger. Each frame from the first headset holds one RGB 1408×1408 image and two monochrome 640×480 images; each frame from the second holds two monochrome 1280×1024 images. Recordings from the first also include scene point clouds from SLAM and eye gaze signal.

Subjects performed pick-up, observe and put-down actions plus actions typical of a kitchen, office and living room. All scenarios were captured in the same motion-capture lab, with lighting, furniture and decorations regularly randomized. The paper reports 425 recordings of around 2 minutes each, with up to 6 objects per recording.

Per-frame ground-truth poses come from a motion-capture system using small optical markers on hands and objects. Hand annotations are in the UmeTrack and MANO formats, and objects are 3D meshes with PBR materials from an in-house scanner.

Of 1.5M frames, 1.16M are fully annotated and passed the authors’ visual inspection. All frames are released with a mask of the valid ones, and the paper says annotations in some frames may be missing or of lower quality. Ground truth is public only for the training split; test-split ground truth is reachable only through evaluation servers.

The paper demonstrates 3D hand tracking, model-based 6DoF object pose estimation and 3D lifting of unknown in-hand objects, and reports that multi-view methods significantly outperform their single-view counterparts. Those are the authors’ results on their own baselines, and Firsthand quotes no benchmark numbers from them.

HOT3D-Clips is a curated subset of 3,832 clips of 150 frames (5 seconds) each, used in two ECCV 2024 challenges: the BOP Challenge 2024 and the Multiview Egocentric Hand Tracking Challenge, per the project page.

HOT3D takes its ground truth from a motion-capture lab with optical markers and four fixed scenarios. Custom egocentric capture is a different design: paid, consented contributors working to a written spec, with 3D hand pose at 21 joints per hand, calibrated stereo depth and IMUs.

The site does not state object pose labels or marker-based ground truth as a delivered field, and this page makes no accuracy comparison between the two.

When a program needs tasks, environments or objects that a published lab dataset does not cover, specified in writing. Firsthand does not build or train models, and makes no claim that its capture replaces marker-based ground truth. If contact signals matter, ask for the separate pilot-ready tactile glove tier.

Confirm the spec on a Spec Pilot, a small fast first batch. Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks. Billing is per validated hour.

What is HOT3D?

A publicly available egocentric dataset for 3D hand and object tracking, with synchronized multi-view recordings and motion-capture ground-truth 3D poses of hands and objects.

What does HOT3D contain?

Over 833 minutes of recordings of 19 subjects interacting with 33 rigid objects, with hand annotations in the UmeTrack and MANO formats and 3D object meshes.

How was HOT3D ground truth produced?

By a motion-capture system using small optical markers on hands and objects, in a motion-capture lab. Some frames may have missing or lower-quality annotations, per the paper.

Is HOT3D open?

The toolkit is under Apache 2.0. The dataset is released under the HOT3D license agreement, and hand annotations need third-party MANO licence terms. Other terms are Not stated in the sources read.

Does Firsthand use HOT3D?

No. Firsthand is not part of it or affiliated with its authors, and makes no claim that HOT3D uses Firsthand data or hardware or that any model was trained on Firsthand data.

Can Firsthand collect hand-object data to a custom spec?

The site states custom egocentric collection to a written spec, fast and on demand, with 21-joint 3D hand pose, calibrated stereo depth and 200 Hz IMUs, confirmed on a Spec Pilot.