Data labeling, quality verification and dataset validation.

Quality verification is the step between annotation and delivery where someone other than the original annotator checks the work against a defined rule. Dataset validation widens that check from individual labels to the dataset as a whole: are the right conditions covered, is the timing between streams correct, is the calibration sound, and is every episode licensed and de-identified.

Both only mean something if the rule is written down. If a spec does not define what counts as an acceptable attempt at a task, QA has nothing concrete to check footage against, and a “validated hour” starts to mean whatever a reviewer decides on a given day. That is why Firsthand’s acceptance test lives in the skill spec, agreed before capture starts.

A second reviewer covers a sampled or full pass depending on the annotation’s density and stakes. Dense labels such as per-frame hand joints and instance masks are where errors compound, so the review depth is set per label type rather than applied uniformly.

Reviewers are measured on consistent application of the rule, contributors on usable output, and the buyer pays for data that trains rather than footage that has to be filtered later. Because hand pose carries a per-joint visibility flag, a reviewer can check that occluded joints are marked rather than invented — one of the most common silent failures in pose labeling.

Every stream carries its own PTP-domain timestamp, so timing between head video, wrist video, depth and IMU can be verified rather than assumed. Every episode ships its calibration — intrinsics, SE3 extrinsics and the RMS reprojection error, held below 0.28 pixels — so geometric soundness is recorded per episode, not asserted once for the whole dataset.

Coverage is validated against the spec’s condition matrix. Every delivery carries a data card recording which cells were captured and what was rejected, and each episode’s meta.json carries its reject_flags. Consent is validated too: each episode links to a signed contributor release, and faces, plates and screens are detected, tracked and irreversibly blurred at ingest, with a reviewer confirming the result. The de-identification pass fails closed — if a reviewer is unsure a region is fully covered, the episode is held or discarded rather than shipped.

It is rejected, logged with a reason and, where the spec calls for it, recaptured at Firsthand’s cost. It is not billed. A reject log is what lets a buyer audit the bar a provider is actually holding, rather than trusting a summary number.

If the same failure keeps recurring, the answer is often to revisit the spec rather than keep recapturing — for example, if the geometry that must stay in frame is impossible to hold in a particular environment. That decision is made with you, and the data card records it.

Ask to see a reject log with reasons, not a pass rate. Ask who writes the acceptance test and whether it is agreed before capture. Ask whether failed items are billed, and whether calibration and sync are recorded per episode. Ask whether consent records are delivered with the data or only held internally.

On a Firsthand custom collection the answers are fixed: a written acceptance test in the spec, a visible reject log, recapture at our cost, per-episode calibration and timestamps, and consent records delivered with the data. First validated batches arrive in a median of 48 hours where coverage is deep and three to four weeks where it is thin.