How to Turn a Human Job Into Robot Training Tasks

A job title like "warehouse associate" or "line technician" describes hours of varied work: walking, picking, inspecting, packing, recording, handing off. No robot policy is trained on a job title. Policies are trained on tasks with a clear start, a clear end and a consistent set of objects. The practical first step in any physical AI deployment is therefore to re-express a human job as a set of task units a robot can learn one at a time.

Decomposition also makes the business case honest. Some parts of a job are well within reach of current manipulation policies; others require dexterity, judgment or mobility the robot does not yet have. Breaking the job apart shows which units a robot can take on first, which is where early deployments and early data spending should go.

A task unit is the smallest piece of work that still has a meaningful outcome: "pick a part from a bin and place it in a fixture", "fold a towel and stack it", "scan a package and set it on a cart". A good unit is named by verb and object, has an observable start and end, and happens often enough that collecting many examples of it is realistic.

Inside each unit are action segments — reach, grasp, move, place, release. Action segmentation is how those sub-steps are labeled in the data, and it is what lets a model learn the structure of a task rather than a blur of motion. Our guide to action segmentation for robot learning covers the labels in detail.

• Observe the job as it is actually done, in its real environment, and list every distinct thing the person does.
• Group those actions into units that each have a clear start, end and outcome.
• Name each unit by verb and object, e.g. "insert cable into connector" rather than "assembly".
• Note the objects, tools, fixtures and environment conditions each unit depends on.
• Note how each unit fails in practice — drops, misalignments, regrasps — since those become failure-case cells.
• Rank the units by robot readiness and value, and pick the first set to train.

Egocentric video is a useful tool for the first step as well as for training. Watching first-person footage of the job, from the worker’s point of view, surfaces the hand motions and small adjustments that are easy to miss from a doorway.

The ranking is a judgment for the deployment team, not a formula. What matters for data is that each chosen unit becomes a concrete, capturable spec.

Each chosen unit becomes a skill spec: the task, the environment, the object list, the capture setup, the labels and the accept/reject rule, plus a per-cell hour budget across the conditions that matter. Conditions might include object variants, lighting, left- or right-handed approaches, and the failure modes noted during decomposition. The spec is what contributors perform against and what QA checks against.

Once a unit is specced, contributors perform it in matched environments while wearing the capture rig. A spec pilot comes first to confirm the capture and labels are what the training team expects; then the collection scales against the same spec. After a spec is signed, first validated data arrives in a median of 48 hours where contributor coverage is deep and in 3–4 weeks where it is thin.

Because each unit is its own spec, new units can be added as the deployment grows. The first units train the robot for the easiest part of the job; later units extend it as the hardware and model mature. Every delivery comes with a data card and a buyer-owned license, so the data built up over a deployment remains the buyer’s to reuse.