What Are Acceptance Criteria for a Custom Dataset?

Without written criteria, a delivery can only be judged by opinion. A buyer cannot test a claim that data is clean, and a vendor cannot show it was met. Stated criteria fix what is measured, the value it must hold and the consequence, so the delivered files can be checked and the invoice tied to what passed.

Firsthand states a machine check on every episode and human review against the skill spec, with six reject reasons published on its capture page: sync breach, calibration drift, occlusion, invalid depth, consent gap and spec miss. Misses are rejected and recaptured at Firsthand’s cost, the reasons ship in a reject log, and billing is per validated hour. A buyer sign-off step, review window, pass rate and refund terms are Not stated.

Acceptance criteria are the written conditions a delivered dataset must meet before the buyer treats it as complete. Each one names what is measured, the threshold it must meet and what happens to data that misses it. In a custom collection they are agreed before capture, so delivery is judged against the spec rather than against impressions.

Three things: a measure, a threshold and a consequence. A weak criterion such as “clean data” has none of them. A usable one says which signal is checked, the value it must stay within, and whether failing data is rejected, recaptured or left out of the count.

The site’s skill-spec guide asks for a reject rule and one acceptance test a reviewer can apply, alongside the named skill, the visible geometry and the hour budget per condition.

The capture page states a machine check on every episode, then human review against the skill spec the episode was captured for. It lists six reject reasons: a sync breach, calibration drift, occlusion, invalid depth, a consent gap and a spec miss. The first table gives each as the site words it.

Calibration also gates the start of a session: reprojection error must land under 0.28 px or the session does not start, and the drift between opening and closing solves is written into episode metadata.

The site states that rejected episodes are recaptured at the provider’s cost and that the reasons ship with the delivery as a reject log. Billing follows validated hours, so rejected or off-spec footage is not something the buyer pays for.

Faces, plates and screens are blurred and confirmed by a reviewer before anything ships, and each batch data card lists consent artefact IDs per episode.

Two signatures are stated. The accept and reject criteria are signed before capture, and commercial terms sit in an order form or master services agreement signed by both parties. A separate buyer sign-off step on a delivered batch, a review window, a dispute process and credit or refund terms are Not stated, so put them in the order form if you need them.

The published clock for fast collection starts at a signed spec, with a first validated batch in a median of 48 hours in environments where coverage is already deep. Because the clock ends at a validated batch, the criteria in the spec decide when it stops.

List each check with its measure, threshold and consequence, and mark any value you need that the site does not state. Sync, calibration, occlusion and depth have stated figures to start from. Coverage minimums, label accuracy and pass rates do not. Request a custom dataset and send the list with the task.

What are dataset acceptance criteria?

The written pass-or-fail tests a custom dataset must meet before it counts as delivered. Each names a measure, a threshold and what happens on failure.

How are acceptance criteria different from a quality gate?

A quality gate is the check itself. Acceptance criteria are the set of conditions, written into the spec, that those checks enforce.

What does Firsthand do with footage that misses the criteria?

The site states that rejected episodes go to a reject log with reasons, are recaptured at the provider’s cost, and are not billed, because billing is per validated hour.

Does the site state a pass rate or a review window?

No. A pass rate, the share of episodes rejected, a review window, a dispute process and credit or refund terms are Not stated.

Can acceptance criteria be set for a fast, on-demand collection?

Yes. Criteria are written into the signed spec, which starts the published clock: a first validated batch in a median of 48 hours where coverage is already deep.