How to Build a Vetted Workforce for Robotics Data Collection

Egocentric robotics data is performed by people. Every episode is a person wearing a capture rig doing a task in a real room. That means the dataset inherits the properties of the workforce that produced it: how carefully contributors follow the spec, how varied their hands, homes and habits are, whether they consented properly, and whether they were paid fairly for the work.

A dataset gathered from an unscreened, loosely managed crowd tends to fail in predictable ways. Some contributors drift from the spec. Some footage is captured without the consent documentation a commercial buyer needs. Diversity collapses toward whoever was easiest to recruit. The data may look large, but its validated hours — the hours that actually pass QA — can be a fraction of the raw total.

Vetted means a contributor has been screened, onboarded into a consistent flow, and is held to the same standard as every other contributor. At Firsthand, contributors across 150+ countries and 500+ languages and locales are screened and paid through the same QA and consent flow every time, which is what lets an on-demand collection stay consistent across many people and places.

A skill spec is the working contract between the buyer and every contributor. It states the task, the environment, the objects, the capture setup and what counts as an accepted episode. Because every contributor works against the same written spec, two people in different countries produce episodes that are comparable rather than merely similar.

The spec also makes QA fair. When the accept/reject rule is written down, a rejected episode can be explained in terms of the rule rather than a reviewer’s taste. That consistency is what lets a collection grow from a spec pilot to production volume without the quality drifting as more contributors join.

Every episode is checked before it counts. Firsthand reports validated hours rather than raw hours: an hour is counted only after it passes sync, calibration, label and spec checks. Streams must stay inside the 2.0 ms sync reject ceiling, and the published residuals are a 1.12 ms median and 1.94 ms p99. Episodes that fail are logged with a reason, so the reject log becomes part of the delivery record.

Reporting validated hours changes the incentives for everyone involved. Contributors are measured on usable output, reviewers on consistent application of the rule, and the buyer pays for data that trains rather than footage that has to be filtered later.

Trust in a robotics dataset is ultimately trust that the people in it agreed to be in it. A consent chain records that agreement for every contributor, and anonymization removes personal data such as faces and license plates from the footage, with human review on top of automated detection. Our guide to consent and privacy for robot pretraining data covers how that holds up as a collection grows to many people, homes and workplaces.

This guide describes Firsthand’s operating process. It is not legal advice; consult counsel for requirements that apply to your organization.

Diversity in robotics data is mostly diversity in people and places: different hands, different kitchens, different tools, different ways of doing the same task. A workforce spread across many countries can balance activities, environments and regions so a policy does not quietly overfit to one kitchen. Our guide on why dataset diversity matters explains how that breadth shows up in generalization.

• How are contributors screened and onboarded, and is the flow the same for everyone?
• Is there a written accept/reject rule, and can you share a sample reject log?
• Are hours reported as validated or raw?
• Is consent recorded per contributor, and how are bystanders handled?
• How are contributors paid, and through what flow?
• How is diversity across regions, environments and people balanced?