How to Capture Failure Cases in Robot Training Data

A failure case is any part of a task that is not a clean, uninterrupted success: transit between locations, occlusion during a grasp, low light, a dropped object, or a recovery attempt after something goes wrong. It describes what happens within the episode, not whether the episode is usable — a messy middle can still be a validated hour if it meets sync, calibration and consent requirements.

This is a distinct concept from a failed episode in the sense of a rejected one. An episode that includes a dropped object and a recovery attempt is not automatically discarded; it is exactly the kind of episode failure-case coverage is designed to keep rather than edit out. The only episodes that get rejected during QA are ones that fail the underlying checks — broken sync, missing calibration, no consent — not ones that simply contain a hard moment.

Failure-case coverage is the deliberate inclusion of these hard parts in a dataset, in place of curating footage down to only the clean, successful segments. The word "deliberate" matters: coverage that happens by accident, because a collector did not bother to re-shoot a messy take, is not the same as coverage that was planned for in advance and checked for at delivery.

Policies fail in exactly the situations that highlight-reel training data cuts. If a dataset only shows clean grasps in good light with nothing in the way, a model trained on it never sees what recovery looks like, and it breaks the first time reality is messy — which, outside a demo, is most of the time.

That is the difference between a dataset that teaches robustness and one that teaches a best-case demo a policy can only reproduce under the same clean conditions it was trained on. Capturing failure cases on purpose, rather than hoping a collection session happens to include some, is what turns that difference into something a buyer can actually spec and verify rather than leave to chance.

Leaving failure-case coverage to chance also makes it uneven across a dataset in ways that are hard to see until a model is already trained on it. A collection program that is not planning for specific hard cases tends to accumulate the easy ones — a brief occlusion here, a bit of transit there — while the harder, more informative failures like a multi-step recovery after a drop stay rare simply because they take more effort to capture and nobody asked for them by name.

A skill spec is where failure-case coverage gets planned rather than left to chance: naming the occlusion angles, lighting conditions, and recovery scenarios that matter for the target task before capture starts, so those cells count toward validated hours instead of being an afterthought discovered after delivery.

In practice that means being as specific in the spec about failure cases as about the successful task itself. Instead of a generic instruction like "include some failures," a spec names the actual scenarios that matter for the deployment environment — for example, an object slipping mid-grasp and being re-grasped, a hand passing behind another object mid-reach, or a task being attempted in dim ambient light rather than only under bright, even lighting. Each named scenario becomes a coverage cell the same way a successful task variant would, which means it can be counted, tracked, and checked for completeness the same way the rest of the skill spec is.

Specifying failure cases this concretely also makes them easier to verify after delivery. A vague request for "failure diversity" cannot be checked against anything; a spec that names occlusion angles, lighting ranges, and specific recovery scenarios gives a buyer a list to check the delivered dataset against, cell by cell, rather than trusting a general claim about coverage.

Firsthand ships the transitions and hard cases uncut — walking between rooms, exposure hunting in dim light, occluded grasps — because that is what a head-mounted camera actually returns when it is not curated down to highlights. A contributor performing a planned failure-case scenario does not pause, reset, or re-stage the footage; the episode runs through the difficulty and into the recovery exactly as it would during real use, with video, depth, hand pose and IMU still recording the whole way through.

That continuity matters for recovery specifically. A recovery is only informative as training data if the sensor streams that led up to it — the moment of the drop, the occlusion that preceded a misjudged grasp — are present in the same episode as the correction. Cutting a failure case down to only the recovery, or re-staging it as a separate clean take, removes the very context a policy needs to learn why the recovery was necessary in the first place.

Firsthand labels sub-actions with action_segments: {start_ns, end_ns, verb, noun} records with nanosecond timestamps aligned to the same PTP clock domain as video, depth, and hand pose. A recovery attempt is labeled the same way any other sub-action is — as its own segment, with its own start and end boundary — rather than being absorbed into the segment for the step that failed. That keeps the failed attempt and the recovery attempt as two distinct, separately-timed units a policy can learn from independently.

Frame-level fields add detail within those segments. hand_joints and hand_visible track hand pose and occlusion on every frame, so a policy can see exactly when and how a grasp was occluded, not just that occlusion happened somewhere in the segment. contact_events flags the moment contact with an object was made or broken, which is what distinguishes the frame a drop actually occurred from the surrounding motion. instance_masks separate the relevant objects pixel by pixel on frames where the program spec calls for it, which helps localize what a recovery was reaching for.

Together, these fields mean a failure case is not represented as a single flag on the episode but as a structured sequence: an action segment for the moment things went wrong, frame-level labels describing exactly what the hand and scene looked like during that moment, and a separate action segment for the recovery that followed — all timestamped against the same clock so the three line up exactly.

Failure-case coverage is checked the same way any other coverage cell in a skill spec is checked: against the list of occlusion angles, lighting conditions and recovery scenarios named before capture started, not against a general impression of whether the dataset "feels" diverse. A coverage cell either has enough validated hours against it or it does not, and that check applies equally whether the cell describes a successful task variant or a planned failure scenario.

That check sits alongside, not instead of, the underlying quality checks every episode goes through — sync, calibration and consent. A messy episode with a clean recovery still has to pass those checks to count as a validated hour; failure-case coverage is about what is included in the dataset, not a separate bar an episode has to clear to be considered acceptable.

There is no single ratio of failure cases to successes that fits every task, because the right amount depends on which failure modes actually matter for the target deployment. This is exactly why failure-case coverage is planned in the skill spec rather than applied as a fixed percentage: a task where occlusion is rare in deployment does not need heavy occlusion coverage, while a task where drops and recoveries are common does.

In practice, the planning question is not "what fraction of episodes should fail" but "which specific hard cases, named in the spec, need enough validated hours behind them to be learnable." Shallow, well-covered failure-case cells can ship in the same turnaround as the rest of a collection; long-tail failure-case coverage that requires more scenario variety — rarer recovery types, less common lighting or occlusion combinations — takes longer, the same way long-tail coverage of any kind does. See how validated hours are counted for the full breakdown of how coverage depth affects turnaround.