Image and video annotation for robotics training data.

Robotics perception work draws on a small, well-understood family of spatial labels. A bounding box is the smallest axis-aligned rectangle that encloses an object, stored as pixel coordinates �� most often (x, y, width, height) or two corners. It is cheap to produce and good enough for detection, but it says nothing about the object’s actual outline, so a box around a mug also contains the table behind it.

A polygon traces the outline of an object as a list of vertices. It is tighter than a box and is a common interchange format for segmentation labels. A per-pixel segmentation mask goes one step further and assigns every pixel to an object instance or to the background. Instance segmentation distinguishes two identical objects from each other; semantic segmentation only says which class each pixel belongs to.

For manipulation, the difference matters. A grasp policy needs to know where the hand ends and the object begins, and a box cannot express that boundary. That is why Firsthand delivers segmentation as per-pixel instance masks rather than boxes alone. A polygon outline can always be traced from a mask when a pipeline expects polygons, but a mask cannot be recovered from a box without re-labeling.

Most image annotation tooling was built for single photographs: one frame, one set of labels, no relationship to the frame before or after. Egocentric robotics footage is different. The camera is on a moving head or wrist, the hands occlude the object at exactly the moment of contact, and the same object has to be recognised as the same instance from one frame to the next.

Labeling frames in isolation produces masks that flicker, instance identities that swap when a hand passes in front of an object, and boundaries that disagree with the depth geometry. Firsthand annotates against the synchronized depth stream and the hardware clock the video was recorded on, so masks agree with the captured geometry and every label carries the same PTP-domain timestamp as the frame it belongs to.

The head camera records at 3840×2160 and 60 Hz, the two wrist cameras at 1920×1080 and 60 Hz, and depth at 848×480 and 30 Hz. Instance masks are stored at the depth resolution, which is what lets a mask be lifted into 3D alongside hand joints that are expressed in metres in the head-camera frame.

Classification starts before capture, not after. The skill spec names which object classes are in scope, which grasp families and which environments, and that vocabulary is what annotators label against. This keeps labels consistent across contributors and sites, and it means the delivered classes already match what your loader expects.

Tracking links the same instance across frames. In the delivered episode, instance masks and the objects annotation group describe which object is which over time, and an object tracked across frames gives a policy a persistent handle on which object is being manipulated. The same detect-then-track pattern is what the de-identification pass uses on faces and plates: a detector returns a bounding box and a confidence score per frame, and a track id links the same box across frames so coverage does not flicker as a subject moves.

Contact and grasp events are annotated against the depth stream alongside the masks, so a tracked object carries the moment of physical interaction as well as its outline.

A second reviewer covers a sampled or full pass depending on the annotation’s density and stakes. Items that miss are rejected and recaptured at no extra cost, and the reject log — with reasons, not just pass counts — is visible to the buyer.

Because a skill spec names the acceptance test up front, a reviewer can accept or reject a batch against a written rule rather than personal judgment. Billing is per validated hour, so footage that fails QA is not billed. See the data labeling and quality verification page for the full review workflow.

This program annotates video captured to your spec on Firsthand’s synchronized hardware. That is a deliberate constraint rather than a limitation of the tooling: depth-aligned masks, 3D hand pose and contact events are grounded in the same clock and calibration the footage was recorded on, and those do not exist for footage shot on unsynchronized cameras.

If your goal is a fast, on-demand custom collection with segmentation and object labels built in, the skill spec is where you define the class vocabulary and which label types each episode needs. First validated batches arrive in a median of 48 hours where coverage is deep and three to four weeks where it is thin.