What Are Annotation Guidelines for Robot Training Data?

A label is only as consistent as the rule behind it. The site states that raw capture is labelled, keypointed, segmented or event-tagged as the spec requires, and that a generic taxonomy will not match a specific target task. Written guidelines fix the vocabulary and the boundaries before labeling starts, so a second reviewer can check a label against a rule instead of an opinion.

Firsthand states annotation level in the skill spec explicitly rather than leaving it implied. For skill-level labels, the spec names the verb-noun vocabulary and which sub-actions get their own segment boundary. For frame-level labels, it states the required fields, with hand_joints and hand_visible at minimum for any task involving grasping. Delivery is described as labelled, schema-conformant data with the annotation guidelines that produced it. The guideline document’s format, length and versioning are Not stated.

Annotation guidelines, also called labeling guidelines, are the written instructions a labeler and a reviewer both follow: what each label means, where an action segment starts and ends, and which fields must be filled. An annotation spec for egocentric video is the same idea scoped to first-person footage with hands, objects and depth.

They are neither the labels nor the schema. The episode schema lists which fields exist; the guidelines say how a labeler fills them. The site states that a delivery carries the guidelines that produced its labels.

A skill spec is the written contract for one capture program: the named skill, the environments and conditions, the acceptance criteria and the target hours. Annotation level is set inside that same spec. A task taxonomy is different: it is the map of many tasks, from sector to environment to task to skill, used to plan coverage.

Because a generic taxonomy will not match a specific task, the spec names the verb-noun vocabulary the labels use. Action labels, the delivered label types, are what the guidelines govern. Whether the guidelines arrive as a standalone file is Not stated.

Labels are checked against a written acceptance test from the skill spec. The skill-spec guide’s worked example has a held-out reviewer name the verb and noun from the first 15 frames without reading the label. A second reviewer covers a sampled or full pass depending on the annotation’s density and stakes, and review depth is set per label type, since dense labels such as per-frame hand joints and instance masks are where errors compound.

Because hand pose carries a per-joint visibility flag, a reviewer can check that occluded joints are marked rather than invented. Items that fail are rejected, logged with a reason and recaptured at Firsthand’s cost, and are not billed. The reject log with reasons is visible to the buyer, and each episode’s meta.json carries reject_flags. Agreement scores, label accuracy rates and the second-pass sample rate are Not stated.

The EPIC-KITCHENS-100 paper (arXiv 2006.13256) reports that a new annotation pipeline gave denser annotations, 54% more actions per minute, and more complete ones, 128% more action segments, than the earlier version of the dataset. Ego4D (arXiv 2110.07058) describes itself as a massive annotated dataset and benchmark suite. Neither evaluates Firsthand data, so this page attributes no result to it.

Custom egocentric data collection starts from a written skill spec, usually proved by a Spec Pilot before volume scales. Where coverage is deep, a signed spec can reach a validated first batch in a median of 48 hours; where contributors must be recruited first, the site gives three to four weeks. Send the task, your verb-noun vocabulary, the label fields you need, your delivery format and your review bar, then ask to see the guidelines on the pilot batch.

What are annotation guidelines?

They are the written rules a labeler and reviewer follow: the vocabulary, where labels start and end, and which fields are required.

Are annotation guidelines the same as a skill spec?

No. A skill spec is the written contract for a capture program. Annotation level, including the verb-noun vocabulary and required fields, is set inside it.

Does Firsthand deliver the guidelines with the data?

The site describes delivery as labelled, schema-conformant data with the annotation guidelines that produced it. The file format is Not stated.

How does Firsthand check labels?

Against a written acceptance test, with a second reviewer on a sampled or full pass. Failures are rejected, logged with a reason and recaptured at Firsthand’s cost.

Does Firsthand state an agreement score or accuracy rate?

No. The egocentric pages state a second-reviewer pass but no inter-annotator agreement score, accuracy rate or sample rate.

Can I set my own action vocabulary?

Yes. The skill spec sets the action vocabulary, environment and label set before capture, so labels use the terms your training stack expects.

Which label fields are in the episode schema?

The annotations group lists hand_joints, hand_visible, body_joints, contact_events, instance_masks, objects, action_segments and gaze_target.