Glossary
What Is Workflow Capture for AI Training?
Last reviewed: October 2026
Short answer
Workflow capture is recording a real enterprise workflow or standard operating procedure as first-person (egocentric) video and synchronized data, then structuring it for AI training: the workflow is written into a skill spec, broken into task units through task decomposition, and labeled with action segments and hand pose across captured episodes, delivered under a buyer-owned commercial license in a training-ready format.
01
Why it matters
A workflow as it exists today — a written SOP, a training video, or know-how in an experienced worker’s hands — is not yet something a model can learn from. It has to be turned into a written skill spec that states which tasks, scenes, and streams to capture and what counts as an acceptable take, then broken into step-sized task units so a model is trained on repeatable units rather than a whole job at once. Workflow capture is also distinct from passively recording an existing activity: it is a first-person, hands-on recording of a physical task, not a recording of on-screen software activity, and it is captured against your own spec and failure modes rather than bought as a generic, off-the-shelf dataset captured for someone else’s environment.
02
How Firsthand handles it
Firsthand turns a workflow or SOP — whatever form it exists in today — into a written skill spec, decomposes it into task units through task decomposition, and captures it as episodes labeled with action segments marking each step’s boundaries and hand pose where the spec calls for it. Where a step is prone to going wrong, failure case coverage captures the error and the worker’s recovery as its own labeled segment rather than only the clean completion path. Every contributor signs a documented release as part of a consent chain, and faces, license plates, and screens caught incidentally in frame are detected and irreversibly blurred, with a human reviewer confirming the result before anything ships. A workflow capture program typically starts with a Spec Pilot — a small, fast first batch QA’d against the written accept criteria — before scaling to full volume, with delivery in RLDS, LeRobot, WebDataset, HDF5, zarr, or Rerun .rrd under a perpetual, buyer-owned, non-exclusive license by default.
03
What workflow capture records, by element
| Workflow element | How it is captured or labeled | Delivered as |
|---|---|---|
| Workflow or SOP | Written into a skill spec before capture starts | Skill spec document |
| Task unit | Identified through task decomposition | Named step with a completion check |
| Step boundary | Marked by action segments within each captured episode | Action-segment labels: start/end timestamps, verb-noun |
| Grip or manipulation | Captured as hand pose where the spec calls for it | Hand pose labels or stream |
| Error and recovery | Captured uncut as its own labeled segment via failure case coverage | Labeled failure/recovery segment |
| Consent | Documented release signed by every contributor | Consent chain record |
| Bystanders and screens | Detected and irreversibly blurred, confirmed by human review | Anonymization record |
05
Workflow capture vs screen recording vs generic off-the-shelf video
| Workflow capture | Screen recording | Generic off-the-shelf video | |
|---|---|---|---|
| What it captures | A real, hands-on physical workflow from a first-person view | On-screen software activity only | Similar-looking tasks captured for a different buyer’s spec |
| Structure | Written skill spec, task decomposition, action segments | No task structure — raw screen activity | Whatever structure the original seller applied, if any |
| Fit to your job | Matches your own objects, environment, and documented failure modes | Captures software use, not physical manipulation | Captured against someone else’s environment and failure modes |
| License | Perpetual, buyer-owned, non-exclusive by default | Not applicable | Varies by seller, often shared and non-exclusive |
06
Frequently asked questions
- What is workflow capture?
- Recording a real enterprise workflow or SOP as first-person video and synchronized data, then structuring it for AI training through a written skill spec, task decomposition, and action-segment and hand pose labels.
- How is a workflow turned into a skill spec?
- The workflow — an SOP document, a training video, or tacit knowledge in a worker’s hands — is described in terms a capture program can act on: which tasks make it up, which scenes and variants to cover, which streams to capture, and the accept criteria a take has to clear. That description becomes the written skill spec.
- How is workflow capture different from screen recording?
- Workflow capture is a first-person recording of a hands-on physical task, not a recording of on-screen software activity. See the comparison table above for how the two differ in structure and license.
- How is workflow capture different from buying a generic dataset?
- A generic, off-the-shelf dataset was captured against someone else’s spec, environment, and failure modes. Workflow capture is captured against your own written spec, so the objects, environment, and documented failure cases match the job a model will actually be deployed on.
- What data does workflow capture deliver?
- Episodes labeled with action segments and, where specced, hand pose, plus any failure and recovery segments called for by the spec, delivered in a training-ready format such as RLDS, LeRobot, WebDataset, HDF5, zarr, or Rerun .rrd.
- How is consent handled during workflow capture?
- Every contributor signs a documented release as part of a consent chain, and that record travels with the data through delivery.
- How are bystanders and screens protected during workflow capture?
- Faces, license plates, and screens caught incidentally in frame are detected and irreversibly blurred at ingest, with a human reviewer confirming the result before anything ships.
- How do you start a workflow capture program?
- Describe the workflow or SOP and the tasks, scenes, and streams your training pipeline needs. That becomes a written skill spec, and the program typically starts with a Spec Pilot — a small, fast first batch QA’d against the same accept criteria a full order gets — before scaling to full volume.