Guide
Multi-Layer QA for Video Datasets
Short answer
Multi-layer QA on a video dataset means checking quality at each stage rather than once at the end: sync error is validated against a fixed ceiling during capture, coverage (including failure cases) is checked against the capture spec, and consent is verified per participant before a batch is counted as delivered — not inspected retroactively after the fact.
Last reviewed:
Why check quality in layers instead of once at the end?
A single quality check after delivery can only reject or accept a whole batch — it can’t catch a sync drift during a specific clip, a coverage gap in one skill variation, or a missing consent record for one participant, until the footage is already recorded. Checking each of those signals at the stage where it actually occurs catches the problem while it is still cheap to fix, rather than after a full collection has shipped.
What are the layers, and what does each one check?
- Capture-time sync check
- Cross-stream drift between video, depth, and pose is measured against a fixed millisecond ceiling as footage is recorded, using a hardware-triggered clock domain rather than software timestamps.
- Coverage check against the spec
- Recorded footage is checked against the capture spec’s required skill variations and failure-case coverage — stalls, slips, and corrections — not only clean successful demonstrations.
- Validated-hour check
- Only footage that passes the sync and coverage checks counts as a validated hour toward the delivered total; footage that fails is logged and excluded rather than silently kept.
- Consent check
- Each participant’s consent is verified and tied to the specific footage they appear in before that footage is included in a delivered batch.
- Format and license check
- Delivery is checked against the requested training format (RLDS, LeRobot, WebDataset) and the buyer-owned license terms before a batch ships.
Five layers, each checking a different failure mode at the stage where it actually happens.
How is this different from a buyer-side quality checklist?
The robotics dataset quality checklist is what a buyer runs against a finished sample pack or delivered batch to verify those same signals from the outside. Multi-layer QA is how those signals get produced correctly in the first place, at each stage of collection, so that a buyer running the checklist against the delivered data finds documented answers rather than gaps. The two are complementary: one is the production-side process, the other is the buyer-side verification of its output.
What happens to footage that fails one of these checks?
Footage that fails the sync or coverage check at capture time is logged in a reject log and excluded from the validated-hour count rather than delivered anyway. That reject log is part of what a buyer can review under the robotics dataset quality checklist’s validated-hours item, since it shows exactly how much footage was recorded versus how much actually passed.
FAQ
Multi-Layer QA for Video Datasets, answered.
01What does "multi-layer QA" mean for a video dataset?
Checking quality at each stage of collection — sync at capture time, coverage against the spec, consent per participant, format and license at delivery — instead of a single pass/fail check on the finished batch.
02Is multi-layer QA the same as the quality checklist?
No. The checklist is what a buyer runs against delivered data to verify those signals from the outside. Multi-layer QA is the production-side process that produces those signals correctly in the first place.
03What happens to footage that fails a layer?
It is logged in a reject log and excluded from the validated-hour count rather than delivered as part of the batch.
04Why check sync error during capture instead of after?
A hardware-triggered clock domain lets drift be measured against a fixed ceiling as footage is recorded, so a sync problem in one clip can be caught and addressed immediately rather than discovered across an entire delivered dataset.
05Does multi-layer QA cover consent as well as technical quality?
Yes. Consent is verified per participant and tied to the specific footage they appear in before that footage is counted as part of a delivered batch, alongside the sync, coverage, and format checks.
Check it against the sample pack.
40 episodes across 4 environments, delivered in the exact schema these guides describe.