Failure and Recovery Demonstration Data for Robots

It is first-person video of a human performing a task in which the messy moments stay in. On this site a failure case is any part of a task that is not a clean, uninterrupted success: transit between locations, occlusion during a grasp, low light, a dropped object, or a recovery attempt after something goes wrong. Failure and recovery demonstration data is a delivery built around those moments, with the recovery labeled as its own step.

A failure case describes what happens inside the episode, not whether the episode failed quality validation. An episode with a dropped object and a clean recovery can still be a validated hour if it meets the sync, calibration and consent checks.

This page is about buying that data, not a how-to, an explainer or a glossary definition; those are linked below. The data is human demonstration recorded from the performer’s own viewpoint. It is not robot or teleoperation data, and this page makes no claim that either is captured.

Policies fail in exactly the situations that highlight-reel training data cuts. If a dataset shows only clean grasps in good light with nothing in the way, a model trained on it never sees what recovery looks like, and it breaks the first time reality is messy. Outside a demo, that is most of the time.

That is the difference between a dataset that teaches robustness and one that teaches a best-case demo. The site’s failure-case guide adds that leaving coverage to chance makes it uneven: a program that is not planning for specific hard cases tends to accumulate the easy ones, while harder failures, such as a multi-step recovery after a drop, stay rare because nobody asked for them by name.

Buying the data on purpose makes it specifiable and verifiable. You name the hard cases, they become coverage cells with validated hours behind them, and you check the delivered set against that list. This page does not state failure rates or coverage targets, because the right amount depends on which failure modes matter for your deployment.

The skill spec is where failure coverage is planned. It names the occlusion angles, lighting conditions and recovery scenarios that matter for the target task before capture starts. A generic instruction like “include some failures” cannot be checked against anything. A spec that names the scenarios can.

The failure-case guide gives examples: an object slipping mid-grasp and being re-grasped, a hand passing behind another object mid-reach, or a task attempted in dim ambient light rather than only under bright, even lighting. Each named scenario becomes a coverage cell the same way a successful task variant would, so it is counted, tracked and checked like the rest of the spec.

Firsthand designs the first draft of a skill spec with you on a call, including hours budgeted per condition cell, and reports fill rates against it per batch. A Spec Pilot, a small and fast first batch, is reviewed against every axis before volume scales, with the same reject log and validated-hour billing as a full collection, so a gap in failure coverage is caught in the pilot.

Firsthand ships the transitions and hard cases uncut: walking between rooms, exposure hunting in dim light, occluded grasps. That is what a head-mounted camera returns when footage is not curated down to highlights. In the guide’s words, a contributor performing a planned failure-case scenario does not pause, reset or re-stage the footage. The episode runs through the difficulty and into the recovery, with video, depth, hand pose and IMU recording the whole way through.

That continuity matters for recovery specifically. A recovery is only informative if the streams that led up to it, such as the moment of the drop, are in the same episode as the correction. Cutting a failure down to only the recovery, or re-staging it as a separate clean take, removes the context a policy needs.

The setup is the one the rest of the site describes. A head-mounted camera keeps the hands and object in frame from the operating viewpoint, and a wrist-mounted camera can run alongside it for a close-range view of the grasp. Where manipulation precision requirements are high, a stereo pair adds binocular depth. Head and wrist IMU streams share the clock. All streams are hardware-triggered from a single PTP grandmaster clock and aligned under 2 ms, with per-stream nanosecond timestamps in the PTP domain. Capture takes place in the settings the site already lists, not at a customer’s own facility.

A failure is not a single flag on the episode. It is a structured sequence on one clock: an action segment for the moment things went wrong, frame-level labels describing the hand and scene during it, and a separate action segment for the recovery that followed. The table below lists each element and its field.

Action segments are {start_ns, end_ns, verb, noun} records with nanosecond timestamps on the same PTP clock domain as video, depth and hand pose. A recovery is its own segment, not absorbed into the segment for the step that failed. hand_joints and hand_visible track hand pose and occlusion on every frame, and contact_events flags when contact with an object was made or broken, which distinguishes the frame a drop occurred from the surrounding motion.

The fields documented on this site do not include a success or failure outcome flag, a failure type taxonomy or force, pressure or tactile channels, and this page does not claim any. The failure is carried by the structure of the labels above, and each episode ties back to the named skill in your spec through the skill_id field in its metadata.

Failure coverage is checked the way any other coverage cell is: against the list of occlusion angles, lighting conditions and recovery scenarios named before capture started, not against an impression of whether the dataset feels diverse. A cell either has enough validated hours against it or it does not. That check sits alongside the underlying quality checks every episode goes through, not instead of them.

Acceptance is measured against signed accept criteria written into the spec before capture begins. Each batch goes through per-batch QA and a reject log ships with every batch. An episode is accepted only if RMS reprojection stays under 0.28 px and the drift is within tolerance; otherwise it goes to the reject log. Only footage that clears the criteria is billed as a validated hour, and rejected items are recaptured at our cost.

The only episodes rejected are those that fail the underlying checks, such as broken sync, missing calibration or no consent. An episode is not rejected for containing a hard moment.

Every contributor signs a documented consent release, and each episode links to that signed release through a documented consent chain. Faces, license plates and screens are detected and irreversibly blurred at ingest, and a reviewer confirms the result before anything ships. The anonymization record is included with delivery.

Every custom collection ships under a perpetual, buyer-owned commercial license. Exclusivity is available so the data is never resold. The site does not make certification claims for this data, and this page makes none.

Delivery formats are RLDS, LeRobot and WebDataset, plus HDF5, zarr and Rerun .rrd for full-fidelity review of the aligned streams. The episode schema is the canonical layout and each format is a projection of it, so the action segments, contact events and hand labels that mark a failure are part of the layout your loader reads. In WebDataset, each sample’s ann.json carries hand_joints, action and contact_events. Converter source is readable, so you can retarget the schema to your own loader.

Delivery uses the same units as the rest of the schema: metric, with depth in millimetres and hand joints in metres.

Describe the task, the hard cases your policy keeps hitting and the settings it will operate in. We turn that into a written spec covering tasks, environments, capture tier, annotation depth, the named failure and recovery scenarios and the accept criteria, then quote reach, timeline and price against that document before any capture begins. Pricing details are on the pricing page rather than here.

Where coverage is already deep, the first validated batch lands in a median of 48 hours from signed spec. Thin environments typically take three to four weeks, because vetting operators there is the bottleneck, and we quote that timeline up front. Long-tail failure coverage that needs more scenario variety takes longer than shallow, well-covered cells. On-demand top-ups run against the same spec and accept criteria, so new failure and recovery data stays consistent with what you already hold.