Egocentric Hand-Object Contact Data

It is first-person video in which the moments of hand-object contact are labeled as data. The label is called contact_events. It marks when fingers make and break contact with an object, as a timestamped onset and offset against the same clock as the rest of the episode. The video is recorded from the performer’s own viewpoint, so the contact is seen from roughly where a robot’s head camera would be.

This page is about contact events as a label product: a delivery in which the contact timing is the thing a buyer is specifying. It is narrower than hand-object interaction data in general, which covers hand pose, contact and depth together, and narrower than hand tracking data, which centres on 21-joint hand pose per hand. Those pages are linked below, and the comparison table shows how they differ.

Contact events are timing labels. The fields documented on this site do not include force, pressure or tactile channels, and this page does not claim any.

Manipulation is organized around contact. A policy that is learning to pick something up needs to know when the hand arrives at the object, when the grasp is established, and when it is released. Video alone shows a hand near an object, and a single frame cannot say whether the fingers are touching it. The site’s own definition of hand-object interaction data makes the same point: the goal is that a manipulation policy can learn the approach, grasp and release rather than just seeing a hand near an object in a single frame.

Contact timing is also a natural boundary for segmenting behavior. The onset of contact is when a reach turns into a grasp, and the offset is when a placement is complete. Having those boundaries as labels, rather than leaving them for a downstream team to estimate, gives a training pipeline a consistent, spec-defined notion of when a grasp begins and ends across every episode in a dataset.

Because the labels are annotated against captured depth geometry and not guessed from a single frame, the contact moment reflects what the sensors recorded. That is the claim this site makes for its contact events, and it is the reason to buy contact labels as part of a custom collection rather than add them later from video alone.

The field is contact_events. In the episode schema it sits in the annotations group alongside hand_joints, hand_visible, body_joints, instance_masks, objects, action_segments and gaze_target. In the WebDataset format, each sample’s ann.json carries hand_joints, action and contact_events.

Contact events are described in two ways across the site: as contact onsets and grasp events on the hand-object interaction page, and as a timestamped onset and offset against the same clock that marks when fingers make and break contact with an object on the hand tracking page. Granularity is therefore event-level, with an onset and an offset for each contact, rather than a continuous per-frame signal.

The site does not break contact events down into contact types, and it does not state a per-finger or per-hand attribution scheme beyond describing fingers making and breaking contact with an object. On the hand tracking page the field is listed as optional, so contact events are something you include in the written spec rather than assume. If your policy needs contact tied to a particular hand or object, say so when you write the spec and the annotation depth is set against it.

A head-mounted camera keeps the hands and object in frame from the operating viewpoint. A wrist-mounted camera can run alongside it for a close-range view of the grasp, which is useful when fine manipulation matters more than the full-arm reach, and which keeps the contact point in view when the hand and arm would block the head camera.

Depth gives the contact events a measured basis. Where manipulation precision requirements are high, a stereo pair adds binocular depth for finer contact-level geometry. Head and wrist IMU streams are synchronized to the same clock, and the site lists motion data among the signals used for motion and contact event detection.

All streams are hardware-triggered from a single PTP grandmaster clock and aligned under 2 ms, with per-stream nanosecond timestamps in the PTP domain so the synchronization can be verified independently. Distance-invalid pixels beyond the depth sensor’s reliable range are flagged rather than interpolated.

Each label answers a different question. hand_joints gives where the hand is: two hands, 21 joints each, per frame, in metres in the head-camera frame. hand_visible says which joints were actually observed, so occluded joints are flagged instead of guessed. contact_events says when the hand touched the object. action_segments says what the performer was doing, as a start and end timestamp with a verb and noun.

They are annotated against the same clock, so they can be joined without alignment work. A contact onset falls inside an action segment such as a grasp, the hand pose around it shows how the fingers closed, and the visibility flags show where that pose is reliable. A policy that conditions on all four sees the motion, the contact and the intent together.

Everything is organized by episode. The episode schema is the canonical layout every delivery format is a projection of, with meta, streams, annotations and calibration groups. The meta group carries fields such as episode_id and skill_id, so contact-labeled episodes tie back to the named skill in the spec.

Failure cases are recorded as part of the episode instead of being cut out. For a dropped object, the drop and what follows are recorded continuously, contact_events flags the loss of contact, and a new action segment marks the recovery attempt as its own labeled sub-action. For a re-grasp or restart, the attempt stays in the same continuous episode as the failure that preceded it.

Occlusion is handled by hand_visible, which flags occluded joints per frame while the surrounding reach stays inside one action segment. Missed grasps and dropped items are captured and action-segmented alongside clean takes, and they count toward validated hours like any other accepted footage. The full treatment is in the failure-case guide linked below.

Acceptance is measured against signed accept criteria written into the spec before capture begins. Each batch goes through per-batch QA, and a reject log ships with every batch. You are billed per validated hour, so only footage that clears the criteria is billed.

Rejected items are recaptured at our cost. A Spec Pilot, a small and fast first batch, lets you check how contact events are defined and how they line up with hand pose and action segments before you scale volume. The pilot is where you confirm that the labeled onsets and offsets match what your policy needs.

Every contributor signs a documented consent release, and each episode links to that signed release through a documented consent chain. Faces, license plates and screens are detected and irreversibly blurred at ingest, and a reviewer confirms the result before anything ships. The anonymization record is included with delivery.

Every custom collection ships under a perpetual, buyer-owned commercial license. Exclusivity is available so the data is never resold. The site does not make certification claims for this data, and this page makes none.

Delivery formats are RLDS, LeRobot and WebDataset, plus HDF5, zarr and Rerun .rrd for full-fidelity review of the aligned streams. Because the episode schema is the canonical layout and each format is a projection of it, the contact_events annotation is part of the layout your loader reads. In WebDataset it is carried in each sample’s ann.json. Converter source is readable, so you can retarget the schema to your own loader.

Delivery uses the same units as the rest of the schema: metric, with depth in millimetres and hand joints in metres.

Describe the objects, tasks and settings your policy needs contact labels for, and whether contact events should be in scope. We turn that into a written spec covering tasks, environments, capture tier, annotation depth and accept criteria, then quote reach, timeline and price against that document before any capture begins. Pricing details are on the pricing page rather than here.

Where coverage is already deep, the first validated batch lands in a median of 48 hours from signed spec. Thin environments typically take three to four weeks, because vetting operators there is the bottleneck, and we quote that timeline up front. On-demand top-ups run against the same spec and accept criteria, so new contact-labeled data stays consistent with what you already hold.