Glossary
What Is Hand-Object Interaction (HOI) Data?
Last reviewed: October 2026
Short answer
Hand-object interaction (HOI) data is first-person video and labels that record how hands approach, grasp, manipulate and release objects: hand pose, contact, grasp and object state over time. It shows a robot learner not only that a hand is near an object, but how the hand is shaped around it and when contact starts and ends.
01
Why it matters
A manipulation policy learns from the moment fingers meet an object. Video alone leaves that moment ambiguous, because a hand moving near an object looks much like a hand touching it. HOI data resolves the ambiguity by recording hand shape, contact and the manipulated object together, so a buyer can train on the approach, the grasp and the release instead of a caption describing the task.
02
How Firsthand handles it
Firsthand records the signals HOI data is built from during capture, on one hardware clock: 3D hand pose, contact events, grasp events, depth and, where a spec calls for it, tracked instance masks. It collects them fast, on demand and to a written skill spec. Label types the site does not state, such as object pose and affordances, are marked “not stated” below.
03
What is hand-object interaction (HOI) data?
Hand-object interaction data describes one thing: what a hand does to an object. It covers the approach, the grasp, the manipulation and the release, recorded from the actor’s own viewpoint. The signals usually include hand pose, contact, grasp and the state of the object over time.
The term comes from computer vision and robotics, and “HOI dataset” or “hand-object interaction dataset” refers to a collection of such recordings. In egocentric form the camera is worn on the head, so the footage is close to the view a robot’s own head camera would have.
04
How is HOI data different from hand pose estimation?
Hand pose estimation recovers 3D joint positions for every frame. It describes the hand and nothing else, and a skeleton alone does not say whether the hand is touching anything. A contact event adds a moment: when and where a hand made or broke contact with an object. HOI data combines both with grasp information and the object involved, across a whole episode. The first table compares the three.
05
What labels and streams make up HOI data?
A useful HOI recording keeps its streams on one clock: video, depth and hand pose, plus contact and grasp labels and an identifier for the manipulated object. The second table lists each label type against what the Firsthand site states. Several common HOI labels are not stated, and the table says so.
06
Why does HOI data matter for robot manipulation training?
Many manipulation errors happen around contact: a grasp that closes early, a release that comes late, an object that slips. A policy needs examples that show those moments in context. Viewpoint matters too. A head or wrist view shows hand and object geometry as an embodied camera would, which third-person video does not.
HOI data from people carries no robot actions, so using it for a specific robot still means bridging the embodiment gap.
07
How do you scope a custom HOI data collection?
Start from the behavior that fails. Name the objects, grasps and environments the policy must generalize across, then decide which labels it needs: pose, contact, grasp events or masks. A written skill spec records those choices, and a Spec Pilot, a small first batch, tests the spec before the program scales. Where coverage is deep, a first validated batch lands in a median of 48 hours from signed spec.
08
What HOI data does Firsthand capture?
The site describes a head-mounted global-shutter 4K/60 camera, two 1080p/60 wrist cameras, an active-stereo depth sensor at 848×480/30 and 200 Hz 6-DoF IMUs, hardware-triggered onto one PTP clock domain. Hand pose is 21 keypoints per hand in 3D with a per-joint visibility flag, and contact events flag grasp and release.
Delivery is RLDS, LeRobot, WebDataset and HDF5 under a perpetual, buyer-owned license. Custom collections are QA-checked against the written skill spec before delivery, and consent is tracked through a signed consent chain. The site states no object pose, no affordance labels and no force or tactile data.
09
Hand pose, contact event and HOI data compared
| Data | What it records | Time scale | What it cannot show alone |
|---|---|---|---|
| Hand pose estimation | 3D positions of hand and finger joints | Every frame | Whether the hand touches anything |
| Contact event | When and where a hand made or broke contact with an object | Moments: onset and release | How the hand is shaped |
| HOI data | Hand pose, contact, grasp and object state together | A whole episode, approach to release | Force or tactile signals |
11
HOI label types and what the Firsthand site states
| Label type | Why HOI needs it | What the Firsthand site states |
|---|---|---|
| 3D hand pose | Hand shape around the object | hand_joints: 21 keypoints per hand, 3D, metric; per-joint visibility flag |
| Contact events | Where contact starts and ends | contact_events: when and where a hand made or broke contact |
| Grasp events | Where a grasp begins and ends | Listed with contact onsets as interaction labels |
| Action segments | Which sub-action the interaction belongs to | start_ns, end_ns, verb, noun |
| Object identity | Which object is held | instance_masks: tracked object ids, linked to an objects list and to contact events |
| Depth | Approach direction and distance | Active stereo, 848×480/30, metric, per-frame confidence map |
| Grasp type | Hand configuration class | Named on one collection page; not a field in the episode schema |
| Object pose (6-DoF) | Where the object is in space | Not stated |
| Affordance labels | Where an object can be grasped | Not stated |
| Force or tactile data | How hard an object is held | Not stated; contact events are labels, not force data |
12
Frequently asked questions
- What is hand-object interaction data in simple terms?
- First-person video and labels that record how hands approach, grasp, manipulate and release objects, including hand pose, contact and object state over time.
- What does HOI stand for?
- Hand-object interaction. An HOI dataset is a collection of recordings and labels describing how hands act on objects.
- Is HOI data the same as hand pose data?
- No. Hand pose tracks the joints of the hand. HOI data adds contact and grasp events and the object involved, so it describes the interaction rather than only the hand.
- Which HOI labels does Firsthand state?
- 3D hand pose with per-joint visibility, contact events, grasp events, action segments and, where a spec calls for it, tracked instance masks, plus depth. Object pose, affordance labels and force or tactile data are not stated.
- Does Firsthand deliver force or tactile data?
- Not stated. Contact events flag when and where contact changed; they are labels, not force, pressure or tactile readings.
- Can HOI data be scoped to specific objects or grasps?
- Yes. A written skill spec names the objects, grasps and environments, and capture and annotation are built around it.
- How fast can custom HOI data be collected?
- Capture is on demand against a written spec. Where coverage is deep, a first validated batch lands in a median of 48 hours from signed spec; where contributors must be recruited first, three to four weeks is realistic.
Related terms