Glossary
What Is Inter-Annotator Agreement?
Last reviewed: October 2026
Short answer
Inter-annotator agreement is a measure of how consistently different annotators give the same label to the same item. High agreement suggests the guidelines are clear; low agreement signals ambiguity. Firsthand states a second-reviewer pass for egocentric labels, but states no agreement score, metric or target for them. Agreement targets are stated only for expert-written text.
01
Why does label consistency matter for robot training data?
A model trained on labels that two annotators would apply differently learns the disagreement as if it were signal. Consistency matters most where labels are dense: the site notes that per-frame hand joints and instance masks are where errors compound. It also lets a buyer trust a delivery without relabeling it.
02
How is agreement checked on egocentric video labels?
The site does not state an agreement calculation for egocentric labels. It states a review process instead: labels are checked against a written acceptance test from the skill spec, and a second reviewer covers a sampled or full pass depending on density and stakes. The agreement statistic, annotators per item, how disagreements are resolved and the second-pass sample rate are Not stated.
03
What is inter-annotator agreement?
Inter-annotator agreement, also called label consistency, compares the labels that two or more annotators assign to the same items. It is reported as a statistic and shows whether a labeling task is well defined: if trained annotators keep disagreeing, the rule they were given is usually the problem.
It differs from accuracy, which compares labels to a trusted reference, and from QA review, which checks work after the fact.
04
How does agreement relate to annotation guidelines and quality gates?
Agreement depends on guidelines. They fix the vocabulary, the segment boundaries and the required fields, so two annotators start from the same rule. The site states that a delivery carries the annotation guidelines that produced its labels.
A quality gate is the pass-or-fail check that follows. The site states labels as one of its gates, with a second reviewer, and describes reviewers as measured on consistent application of the rule. A numeric agreement threshold is not among the gates it states.
05
What does the site state about label review?
Three things are stated for egocentric data. Review is a second reviewer on a sampled or full pass. Depth is set per label type rather than applied uniformly. And because hand pose carries a per-joint visibility flag, a reviewer can check that occluded joints are marked rather than invented. Each episode’s meta.json carries reject_flags.
Agreement targets are stated in one place only: the expert-written text offer, where inter-annotator agreement targets are set in the spec. That does not extend to video, and the egocentric pages list agreement and label accuracy figures as Not stated.
06
What do third-party papers say about annotator agreement?
Benchmark papers report agreement alongside their annotations. The m2sv benchmark (arXiv 2601.19099) reports inter-annotator agreement as a kappa statistic for its human annotators. A video summarization paper (arXiv 2605.09507) models uncertainty that arises from multiple annotators and reports improved robustness under annotation disagreement. Neither covers egocentric robot data or evaluates Firsthand data, so this page attributes no result to Firsthand.
07
How do you request a custom dataset?
Custom egocentric data collection starts from a written skill spec, usually proved by a Spec Pilot before volume scales. Where coverage is deep, a signed spec can reach a validated first batch in a median of 48 hours; where contributors must be recruited first, the site gives three to four weeks. Send the task, your label types, your review bar and your delivery format. If you need an agreement measure, ask on the pilot batch whether it can be set in the spec, because the egocentric pages do not state one.
08
Label QA steps and where the site’s stated deliverables define them
| Step | Where the site states it | Status |
|---|---|---|
| Written acceptance test | Skill spec, agreed before capture | Stated |
| Second-reviewer pass | Sampled or full, depending on density and stakes | Stated |
| Review depth per label type | Dense labels such as per-frame hand joints and instance masks | Stated |
| Visibility check | hand_visible flags checked; occluded joints marked, not invented | Stated |
| Reject log | Logged with reasons, recaptured at Firsthand’s cost, not billed | Stated |
| Agreement statistic | No detail given | Not stated |
| Annotators per item | No detail given | Not stated |
| Resolution of disagreements | No detail given | Not stated |
| Second-pass sample rate | No detail given | Not stated |
10
Label types and the review the site states for each
| Label | Delivered field | Review the site states |
|---|---|---|
| Action segment | action_segments | A held-out reviewer can name the verb and noun from the first 15 frames |
| 3D hand pose | hand_joints | Second reviewer; depth set per label type, the specific depth Not stated |
| Joint visibility | hand_visible | Reviewer checks occluded joints are marked |
| Instance masks | instance_masks | Dense label, where errors compound; per-type depth Not stated |
| Contact events, objects, gaze | contact_events, objects, gaze_target | Field names stated; per-type review Not stated |
| Expert-written text | Not applicable to video | Agreement and review targets set in the spec |
| Agreement score, label accuracy | None | Not stated |
11
Frequently asked questions
- What is inter-annotator agreement?
- It is a measure of how consistently different annotators label the same item, used to judge whether a labeling task is well defined.
- Does Firsthand report an inter-annotator agreement score?
- No. The egocentric pages list agreement and label accuracy figures as Not stated. They state a second-reviewer pass instead.
- How does Firsthand check labels?
- Against a written acceptance test from the skill spec, with a second reviewer on a sampled or full pass depending on density and stakes.
- Are agreement targets ever set?
- The site states that inter-annotator agreement and review targets are set in the spec for expert-written text. It does not state this for egocentric video.
- What happens to labels that fail review?
- Items that fail are rejected, logged with a reason and recaptured at Firsthand’s cost, and are not billed. Each episode’s meta.json carries reject_flags.
- Is agreement the same as accuracy?
- No. Agreement compares annotators with each other; accuracy compares labels with a trusted reference. The site states neither figure for egocentric data.
Related terms