What Are Rejection Criteria for Robotics Training Data?

A rejection criterion is a specific, checkable condition that a captured clip, episode, or recording session must fail before it is excluded from a delivered dataset. It is the opposite framing of an acceptance criterion written into a capture spec: instead of describing what a batch must do to ship, it describes exactly what disqualifies a piece of footage from shipping at all. Firsthand’s quality-assurance checks — sync validation against a fixed ceiling, per-session calibration against a reprojection threshold, coverage against the spec, and consent verification — are the same checks described across the robotics dataset quality checklist and the multi-layer QA guide, just read from the failure side rather than the pass side.

This matters for a buyer evaluating a dataset or a vendor because “we do QA” is not a verifiable claim on its own. “We reject anything with cross-stream sync error above the stated ceiling, log it, and exclude it from the validated-hour count” is. Rejection criteria turn a quality promise into something that can be checked against a spec after delivery — which is also why they belong in a signed collection spec up front, not discovered after a batch arrives.

This page is a criterion-by-criterion reference for what fails a clip and what happens next. For a buyer-facing checklist of everything to verify before accepting a dataset, see /guides/robotics-dataset-quality-checklist. For how multi-layer QA is structured at capture time, see /guides/video-dataset-quality-assurance. For how validation is performed end to end, see /guides/robotics-training-data-validation.

Some of what a rejection criterion checks for cannot be fixed after the fact. A reprojection error that exceeds the calibration threshold means the recorded hand-joint positions do not reliably correspond to real 3D geometry for that session — there is no post-processing step that restores geometric accuracy that was never captured correctly. The same is true for cross-stream sync error above the ceiling: once video, depth, and pose streams are recorded with a timing offset beyond tolerance, no later alignment step can recover frame-exact correspondence between them. Catching these failures at capture time, rather than after a dataset has already been delivered and used for training, is the reason sync and calibration are checked per session and per episode rather than once at the end.

Other criteria, like a missing consent record or an unblurred bystander face, are not technical limitations so much as conditions that must be satisfied before footage is allowed to exist in a deliverable dataset at all, independent of whether the underlying geometry is otherwise usable. Rejecting at the point of failure — rather than shipping a batch and hoping a downstream review catches the gap — keeps a validated-hour count meaning what it claims to mean: footage that actually passed every check, not footage that was recorded and assumed to be fine.

Firsthand measures cross-stream sync error — the residual timing offset between simultaneously captured streams — against a periodic strobe visible to the cameras and detectable in the IMU, per episode, on a hardware-triggered PTP clock domain rather than software timestamps. An episode that breaches the stated sync ceiling is logged in the reject log rather than delivered. Calibration is checked separately: every episode is bracketed by an opening and a closing calibration solve, and intrinsics plus the SE(3) extrinsic transform between cameras are only considered trustworthy if RMS reprojection stays under 0.28 px at both solves and the measured drift between the two solves is within tolerance. An episode that fails either check goes to the reject log instead of shipping with unverified geometry.

For the full calibration procedure and sync-error measurement methodology, see /guides/camera-calibration-procedure and /guides/how-we-measure-sync-error.

A clip can pass sync and calibration and still fail on coverage or label quality. Coverage is checked against the capture spec’s required skill variations and failure-case coverage — stalls, slips, and corrections, not only clean successful demonstrations — and footage that does not meet the spec’s coverage requirements does not count toward the delivered total. Label quality is checked at the frame level: hand_joints are recorded in metres in the head-camera frame, which only means what it claims if the underlying calibration holds, so a frame-level label is checked against calibration and sync before it is considered trustworthy, independent of whether the geometry itself passed.

A frame-level label failure is a wrong or missing per-frame value: an occluded finger reported as visible, a hand-joint position that does not match the depth geometry at that instant, or a mask boundary that does not track the object it is meant to cover. These are checked against calibration and sync rather than accepted on the strength of the annotation alone, because a label can only be as trustworthy as the clock alignment and camera geometry underneath it.

Every contributor and operator signs a documented consent release as part of the capture process, and that record ships with the data rather than being treated as a one-time assurance kept behind the scenes. Faces, license plates, and screens that appear in frame are detected and irreversibly blurred at ingest, with a human reviewer confirming the anonymization before anything ships. A session without a documented consent release, or footage where anonymization has not been confirmed, does not clear the acceptance bar regardless of how clean the sync, calibration, and labels are — consent and anonymization are checked independently of the technical quality signals, not as a substitute for them.

This is also why a consent gap cannot be “fixed” after delivery the way a label error sometimes can be re-annotated: a release that was never signed, or a face that was never confirmed blurred, is not a quality defect to patch — it is a condition the footage never satisfied, so the footage does not ship.

Footage that fails sync, calibration, coverage, or consent checks is logged in a reject log and excluded from the validated-hour count rather than delivered anyway. On custom collection engagements, rejected items are recaptured at the provider’s cost rather than billed to the buyer, which is also why delivery is billed per validated hour — footage that actually passed sync and QA against the spec — and not per raw hour recorded. Whether a specific rejected clip is worth re-capturing versus simply dropped from the batch generally comes down to whether the underlying scene and task can be re-run (a sync or calibration failure during a repeatable task) versus a one-off condition that is no longer available to re-shoot.

The clearest way to make rejection criteria enforceable is to state the acceptance thresholds in the signed collection spec itself, rather than leaving “quality” undefined and discovering the standard after delivery. A spec that states a sync ceiling, a calibration threshold, the required skill variations and failure-case coverage, and the consent and anonymization steps a batch must clear gives a buyer something to check a delivered dataset against — the same approach described in how to write a skill spec and the robotics dataset quality checklist. An unquantified claim like “we check for sync drift” is not verifiable after the fact; a stated ceiling with a reject log behind it is.

This is also where a buyer should ask what happens to footage that fails each check — whether it is re-captured, dropped, or (in a vendor to avoid) silently kept and delivered anyway. A vendor that can show a reject log alongside a validated-hour total is demonstrating the rejection criteria were actually enforced, not just stated.

A validated hour is footage that has already cleared every rejection check — sync, calibration, coverage, consent — so the validated-hour total delivered to a buyer is, by definition, smaller than the raw hours recorded during capture. The gap between raw hours and validated hours is exactly the footage that was logged in the reject log and excluded, which is why validated hours and raw hours are reported as distinct figures rather than treated interchangeably; see /guides/validated-hours-vs-raw-hours for how that distinction is defined and why it matters for comparing vendors.

Rejection also interacts with delivery timelines on custom engagements: because rejected items are recaptured rather than delivered as-is, a batch with an unusually high rejection rate for a given task or scene takes longer to reach its validated-hour target than one that clears QA on the first pass. This is one reason a capture spec that clearly defines coverage and acceptance thresholds up front — rather than relying on judgment calls during review — tends to move through capture and QA with fewer re-shoots.