Egocentric Video with Segmentation Masks

Egocentric video with segmentation masks is first-person task footage delivered together with per-pixel masks that say which object each pixel belongs to. Instance masks keep two identical objects apart, and tracking gives the same object the same ID from one frame to the next. On Firsthand’s rig the masks are the instance_masks field in the annotations group of the episode schema, on the same timeline as the video, depth and hand pose.

This page sells the collection: fast, on-demand, custom collection scoped to your skill spec, with masks delivered beside the video. It is not the definition and it is not a general annotation service. The instance segmentation glossary entry defines the term, and the egocentric video annotation and image and video annotation pages cover labeling. On those pages the annotation applies to footage captured on Firsthand’s synchronized hardware to your spec, not to footage you already own.

The site’s glossary states the case: manipulation is about a particular object, and instance masks let a model attend to the manipulated item through clutter and occlusion and tie contact events to a specific object ID. Tracked across frames, a mask gives a policy a persistent handle on which object is being manipulated.

A mask also says where the hand ends and the object begins, which a bounding box cannot express. The site also notes that clothing, bedding and bags do not track well with propagated masks, so household episodes ship per-frame instance masks for them. The site states these dependencies but does not quantify how much masks change model performance, and this page does not either.

Masks are delivered as instance_masks in the annotations group, listed beside hand_joints, hand_visible, body_joints, contact_events, objects, action_segments and gaze_target. The schema gives the shape as uint16 [T, 480, 848]: one integer instance ID per pixel on the depth grid. The stream list describes the segmentation stream as instance masks over manipulated objects and target surfaces, with tracked IDs across the episode.

The glossary adds that instance_masks are tracked object IDs per frame, linked to an objects list and to contact events. The format pages say whether a given delivery includes masks is a spec decision, so name them in the skill spec and confirm them on the Spec Pilot batch. The first table lists each field and what the site states about it.

They answer different questions. A mask says which pixels belong to which object instance. A bounding box is the smallest rectangle around an object; on this site boxes are used in the detect-track-blur pass that removes faces, plates and screens, while segmentation is delivered as per-pixel masks rather than boxes alone. Hand pose is 21 joints per hand in metres, and action segments are start and end timestamps with a verb and a noun.

The site delivers these together and keeps them distinct. This page claims for masks only what the site states for masks. The 21-joint count and the visibility flag belong to hand pose, and the verb and noun belong to action segments. The site also says a polygon outline can be traced from a delivered mask, while a mask cannot be recovered from a box without re-labeling. The second table sets the four side by side.

The stream list says manipulated objects and target surfaces. The annotation pages add that object, hand and body masks are annotated where the program spec calls for them, so a policy can separate the hand, the task object and the background as distinct masks.

Which classes are in scope is your decision. The skill spec names the object classes, grasp families and environments before capture, so labels use the terms your training stack expects rather than a generic taxonomy. The site does not publish a class list, and it describes instance masks rather than semantic ones: whether a delivery adds class-only semantic masks is not stated and is set per program.

Masks are annotated against the synchronized depth stream and stored at the depth resolution, 848×480, which is what lets a mask be lifted into 3D alongside hand joints expressed in metres in the head-camera frame. The head camera records at 3840×2160 and 60 fps; how a mask on the depth grid maps onto the 4K head frame is not stated and is confirmed in the spec.

Every label carries the same PTP-domain timestamp as the frame it belongs to. The capture page describes one PTP clock domain across the head-mounted stereo pair, both wrist cameras, the depth sensor and the IMUs, with a 1.12 ms median and 1.94 ms p99 against a 2.0 ms ceiling. The IMUs run at 200 Hz on the head and both wrists on that clock; the site does not describe masks being derived from them. The third table lists the streams.

The episode schema is the canonical layout and its annotations group lists instance_masks as uint16 [T, 480, 848]; HDF5 and zarr mirror that layout. In the WebDataset shards, masks and the other episode-schema fields are not itemised, and which fields appear inside a shard’s annotation file is set in the skill spec.

The site does not state how masks are laid out in the LeRobot v2.1 export, in RLDS, or in Rerun .rrd, so name your loader in the spec. A mask stored as an image array is a different object from a LeRobot observation vector, so confirm the shape you load. The fourth table summarizes this.

An episode is validated on the checks the site defines for every episode: sync within the ceiling, calibration solved and within drift tolerance, coverage cells filled and consent documented. For annotation, a second reviewer covers a sampled or full pass depending on density and stakes, failed items are rejected and recaptured at no extra cost, and the reject log with reasons is visible to you. The site publishes no mask accuracy or overlap figure, so none is claimed here.

On consent, the contributor agreement the site describes covers derivative representations, including segmentation masks, and those layers sit under the same consent as the video. Delivered data ships under a perpetual, buyer-owned license, and billing is per validated hour, so footage that fails QA is not billed.

Several mask details are not fixed on this site and are agreed in the skill spec. The last table separates what the site states from what is set per program, so you know what to put in writing before the quote.

Send the tasks, the environments and regions, and the delivery format you need, and name instance masks in the streams and labels list along with the object classes in scope. The site lists RGB as the baseline and the other streams as added where the spec calls for them, and says each added label adds equipment, QA passes and cost, so order what the model consumes. If you have no written spec yet, the skill spec is the document that names the tasks, the labels and the acceptance test.

The order flow is spec, quote, Spec Pilot, then production. Where coverage for your tasks and regions is deep, the median time from signed spec to first validated batch is 48 hours; where it is thin, turnaround runs three to four weeks. There is no list price.