Procedural Task Video Data Collection

Procedural task video data is first-person video of a task performed in a fixed or partly fixed order of steps — recorded from a camera positioned near the worker’s own point of view, a chest mount, head rig, or wrist camera, rather than a fixed overhead or side-angle camera. What distinguishes it from other egocentric task data is the step structure: the procedure is decomposed into named steps with their own boundaries, rather than treated as one undifferentiated action.

The underlying capture, labeling, QA, and licensing process is the same one used across every custom collection on this site: a written skill spec describing the procedure and its steps, paid and consented contributors performing the real task, synchronized sensor streams recorded alongside the video, and per-batch QA before delivery. What changes for procedural data is the emphasis on step order and boundaries in both the spec and the labeling pass.

A good fit is any task performed the same way, in roughly the same order, many times — an assembly sequence, a multi-step prep or build process, a fixed inspection routine, a setup-then-operate-then-shutdown cycle. Each breaks down into specific, nameable steps, which is what a written skill spec is built to describe.

Procedures with a partly fixed order — where some steps can happen in either sequence but others cannot — are also a reasonable fit, as long as the spec states which steps are strictly ordered and which are flexible. A task with no meaningful step structure at all is usually a better fit for this site’s general action-labeled or task-decomposition pages than for a procedural-specific program.

This page is about procedure structure specifically — step order, boundaries, variation, and completion — rather than about an enterprise SOP program, pretraining-scale diversity, or a specific label-format question. For those angles, see this site’s enterprise workflow data and action-labeled egocentric video pages.

A procedural collection starts the same way any custom collection on this site does: you describe the procedure your training pipeline needs covered — what the steps are, what order they happen in, which steps are strictly sequential versus flexible, and which failure or skipped-step cases need to be represented. That becomes a written skill spec naming the step list and order, the setting, the participant mix, capture tier, annotation depth, and accept criteria, quoted for reach, timeline, and price before any capture begins.

This is the same task decomposition process used across the site: a procedure is broken into named task units with a clear start, end, outcome, and object set, each further divided into labeled action segments. The difference with a procedural spec is that the order between those units is itself part of what gets written down and labeled, not just the units themselves.

Many programs start with a Spec Pilot, a small, fast first batch that proves the spec and accept criteria on a limited step set before scaling to full volume. That lets a buyer confirm step boundaries and label quality on part of a procedure before committing to capture across the full sequence.

Raw video alone does not tell a model where one step ends and the next begins, which is why captured footage is broken into the same structure used across every collection on this site: action segments marking the start and end of each discrete step, and episodes grouping a full procedure attempt from start to finish. For a procedure, those action segments are ordered to reflect the sequence named in the spec, so step order is explicit in the delivered labels rather than left implicit in the raw clip.

An assembly procedure, for example, segments into its component steps — position the part, fasten it, check the fit — rather than being labeled as one undifferentiated clip. That step-level structure is what lets a training pipeline learn the individual steps inside a procedure and the order between them, not just recognize that “a procedure happened.”

Failure and recovery cases are part of the same labeled structure as the clean version of a procedure: a skipped step, a repeated step, a step performed out of order, or a redo after a mistake. Representing how a real contributor deviates from and corrects the procedure, not only the version where every step happens once in order, is part of what a written spec can call out as a requirement for a batch.

A completion check — whether the procedure was finished correctly — is labeled as part of that same structure, so the delivered data distinguishes a complete, correct attempt from one that was abandoned or finished incorrectly.

The written skill spec is the control for consistency: it names the procedure’s steps, their order, and their boundaries once, and every contributor and every batch is captured and labeled against that same definition. QA review before delivery checks calibration, sync, and labels against the spec’s accept criteria, so step boundaries labeled in one batch match the definitions used in another.

Variation between contributors on the same procedure — different grip, pace, or sequence for a flexible step — is expected and captured as real variation rather than treated as an error, as long as it falls within what the spec defines as an acceptable completion of the procedure.

The buyer. Every custom collection, including a procedural program, is delivered under a perpetual, non-exclusive commercial license by default, with exclusivity available, so the finished dataset can be trained on and productized without a research-only or unstated licensing arrangement.

Every episode is linked to a signed contributor release under a documented consent chain, and faces and screens anywhere in frame are detected and irreversibly blurred at ingest, with a reviewer confirming the result before anything is delivered. Delivery is billed and licensed against the accept criteria named in the written spec, with rejected items recaptured at no cost to the buyer before anything ships. Pricing is quoted per spec rather than published as a flat rate; see the pricing page for how a quote is built.

Procedural data ships in the same formats available for any custom collection on this site: RLDS, LeRobot, and WebDataset, plus HDF5, zarr, and Rerun .rrd, or raw streams with metadata if your pipeline needs the unprocessed form. Step order and boundaries are carried in the action-segment and episode structure of the delivered data rather than requiring a separate schema. Readable converter source ships alongside the data, so the schema can be retargeted to your own loader.

Describe the procedure, its steps and order, and the failure or variation cases your training pipeline is missing. That becomes a written spec covering the step list, setting, participant mix, capture tier, annotation depth, and accept criteria, quoted for reach, timeline, and price before capture begins.

From there the program runs through the same end-to-end pipeline as every Firsthand collection: contributor recruiting, capture against the spec, per-batch QA, anonymization, and licensed delivery, with on-demand top-ups available afterward against the same accept criteria.