What Is a Contact Event in Robot Learning?

A robot policy learns from pairs of what the scene looked like and what was done in response. In manipulation, the pivotal part of “what was done” is the instant a hand takes hold of an object or lets go of it, and that instant is hard to pin down from video alone, because a hand moving near an object looks much like a hand touching it. A contact label turns that ambiguity into a timestamped fact: as Firsthand’s failure-case guidance puts it, contact_events flags the moment contact with an object was made or broken, which is what distinguishes the frame a drop actually occurred from the surrounding motion.

Firsthand records contact_events during capture, in the same session as the egocentric video, 3D hand pose and depth, rather than inferring contact afterward from footage. The streams come from a head-mounted global-shutter camera plus two wrist cameras and share one PTP clock domain, so a contact event lines up in time with the hand joints, visibility flags and action segments around it. Contact events are collected fast, on demand and to a written skill spec, then delivered with the rest of the episode in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license.

Three labels describe the same stretch of an episode at different resolutions. An action segment is a span of time: a start_ns, an end_ns, a verb and a noun, such as a sub-action that lasts a couple of seconds. Hand pose is a per-frame value: hand_joints holds the 3D position of 21 joints per hand, and hand_visible flags whether each joint was actually seen or occluded. A contact event sits between the two. It is not a span and it is not a pose; it marks a moment inside the segment where contact with an object was made or broken.

The site does not define separate contact event types for grasps, placements or hand-offs. Those readings are something a buyer builds by combining contact_events with the action segment verb and noun around it, not a taxonomy Firsthand ships.

Contact events are recorded during capture, alongside hand pose and depth, to flag when and where a hand made or broke contact with an object. They come from the same session and the same rig as the rest of the episode: a head-mounted global-shutter camera, two wrist cameras that give a close-up view of the hand and object, and a depth sensor, all hardware-triggered onto one PTP clock domain.

Two quality checks sit behind that. Each rig session is checked against a reprojection-error threshold, rms_reproj_px under 0.28, before its frame-level labels are considered trustworthy, and cross-stream sync error is measured as the residual timing offset between streams. Custom collections are also QA-checked against the written skill spec before delivery. The site does not state a contact detection accuracy figure or name a contact annotation tool, and this page does not either.

In the episode schema, contact_events lives in the annotations group, annotations/contact_events, next to hand_joints, hand_visible, action_segments and the other frame-level and skill-level labels. Every stream in the episode carries int64 nanosecond timestamps on one PTP domain, so a contact event can be matched to a video frame, a depth frame or a hand pose sample without resampling guesswork.

The same fields are projected into RLDS, LeRobot format and WebDataset, with HDF5 and zarr also available, and the field mapping is documented so a buyer can retarget it to their own data loader.

A contact event is a label, not a sensor reading. Firsthand does not describe contact events as force, pressure or tactile data, and standard capture uses cameras, depth and IMU streams, so contact events are not read from contact sensors. A separate pilot-ready glove tier records a tactile glove signal beside the video. A contact event tells you that and when contact changed; it does not tell you how hard anything was held.

It is also not a success or failure flag. In the failure-case guidance, contact_events flags the loss of contact when an object slips or is dropped, and a new action segment marks the recovery attempt as its own labeled sub-action, so the contact event is one input to reading a failure rather than a verdict on it.

Custom first-person collection starts with a written skill spec describing the tasks, settings and contributor requirements, and the spec is where contact event expectations are agreed. A Spec Pilot, a small first batch, is the normal starting point before a program scales. Where contributor coverage is already deep, a signed spec can reach a validated first batch in a median of 48 hours; where contributors need to be recruited first, a realistic timeline runs three to four weeks.

Consent is tracked through a signed consent chain, faces and identifying details outside the task are anonymized, and delivery arrives under a perpetual, buyer-owned commercial license. Capture happens in Firsthand’s contributor settings, not at a customer’s own facility. For cost drivers, see the pricing page.

What is a contact event in robot learning?

A contact event is a labeled moment when a hand, or a tool, makes or breaks contact with an object in first-person video. In Firsthand’s episode schema it is the contact_events field, recorded alongside hand pose and depth during capture.

Are contact events force or tactile data?

No. A contact event flags when and where contact was made or broken. Firsthand does not describe it as force, pressure or tactile data, and standard capture does not read it from contact sensors. A separate glove tier records a tactile glove signal.

How is a contact event different from an action segment?

An action segment is a span of time with a start, an end, a verb and a noun. A contact event is a moment inside that span where contact with an object changed. The two are delivered as separate fields on the same clock.

How do contact events relate to hand pose?

Hand pose says where the hand and its 21 joints were on every frame, with visibility flags for occlusion. A contact event says when the hand actually touched or released the object. They are recorded together during capture and delivered side by side.

Can contact events help identify a dropped or slipped grasp?

Yes, as one input. The failure-case guidance notes that contact_events flags the loss of contact when an object slips or is dropped, and a new action segment marks the recovery attempt. The contact event itself is not a success or failure verdict.

Does Firsthand state an accuracy figure for contact events?

No. The site states no contact detection accuracy figure. What it does state is that each rig session is checked against a reprojection-error threshold (rms_reproj_px under 0.28) before frame-level labels are trusted, and that custom collections are QA-checked against the written spec.

How fast can I get a dataset with contact event labels?

A signed spec typically reaches a validated first batch in a median of 48 hours where coverage is already deep, or three to four weeks where contributors need to be recruited first. A Spec Pilot is the normal starting point, and the pricing page covers cost drivers.