Glossary
What Is a Task Taxonomy for Robot Training Data?
Last reviewed: October 2026
Short answer
A task taxonomy for robot training data is a structured hierarchy that organizes what a dataset covers, typically from sector to environment to task to skill. A robot task catalog is a populated taxonomy. It lets a team plan coverage before collection, then audit gaps and balance afterward, instead of counting hours.
01
Why it matters
Buyers rarely lack hours; they lack a clear view of which tasks and environments those hours cover. A taxonomy turns a dataset into a map, so a lopsided or empty branch shows up while it can still be fixed, and a buyer and a collector share one vocabulary for scoping.
02
How Firsthand handles it
Firsthand’s commercial catalog works as a task taxonomy: each task carries a sector, environment, description, skill level, capture setup and status. Collection runs against a written skill spec, and being listed does not mean a task has been captured.
03
What is a task taxonomy?
A task taxonomy is a structured hierarchy that names everything a robot-training dataset is meant to cover, from broad groupings down to specific units of work. A common shape is sector, then environment, then task, then skill. The same list with descriptions and attributes attached is often called a robot task catalog, or a skill taxonomy for data collection.
It is a planning and audit tool, not a data format. Each node can carry a target and a status, so coverage, gaps and balance are visible before and after collection.
04
How is a task taxonomy different from a skill spec?
A taxonomy is the map of many tasks; a skill spec is the written contract for one capture program. The taxonomy shows where a dataset is thin. The spec defines what will be captured to fill that branch: the named skill, the geometry that must stay in frame, hours per condition, the reject rule and one acceptance test. The comparison table sets them side by side.
05
What levels does a robot-training task taxonomy usually have?
Programs choose their own depth, so read the levels table as a working structure, not a standard. Add a level only when it changes the data you need: a barista task and a warehouse task need different scenes, objects and motions, so they belong in different branches.
06
How do you use a task taxonomy to scope a custom data collection?
Walk the tree and mark each branch as needed, nice to have or out of scope. For each needed task, ask whether your model fails there today. Turn the marked tasks into one skill spec each, with a condition budget, then run a Spec Pilot, a small first batch that proves the spec before volume scales. Tasks that are not listed are scoped the same way.
07
How does a task taxonomy help with coverage and QA?
Coverage becomes a per-branch question instead of one hour count. A reviewer can spot an empty or lopsided branch, such as many tasks in one environment and none in another, and decide whether it is a gap or a deliberate choice. For QA, each batch is checked against the spec for its task, so acceptance is applied per task rather than to the dataset as a whole.
08
How does Firsthand organize environments, tasks and skills?
The commercial catalog behind the environments hub is organized as sector, environment and task. Each task records a description, a skill level of Easy, Medium or Hard, a capture setup and a status. Here “skill” is a difficulty level, not the named skill of a skill spec.
The catalog is a starting point for scoping, not a record of delivered data. A task is captured only against a signed spec, and unlisted tasks can be requested the same way. Episodes ship in formats such as RLDS or LeRobot; the taxonomy itself is not a format.
09
Levels of a robot-training task taxonomy
| Level | Question it answers | Example |
|---|---|---|
| Sector | Which industry or domain? | Food & Beverage |
| Environment | Where does the work happen? | Cafe, Bar & Beverage Service |
| Task | What job is done there? | Professional Barista Workflows |
| Skill | Which verb-and-object unit? | Dice an onion (kitchen/knife_dice) |
| Condition | Which variation cell? | Daylight, low light, cluttered, failure cases |
11
Task taxonomy vs skill spec
| Aspect | Task taxonomy | Skill spec |
|---|---|---|
| Scope | Many tasks across environments | One named skill |
| Main use | Plan and audit coverage and balance | Contract for a capture program |
| Contents | Levels, attributes and status | Skill, geometry, condition hours, reject rule, acceptance test |
| Used for | Scoping and gap review | Capture, QA and acceptance |
12
Fields in Firsthand’s commercial task catalog
| Field | What it records | Example |
|---|---|---|
| Sector | Industry grouping | Food & Beverage |
| Environment | Work environment and typical settings | Cafe, Bar & Beverage Service |
| Task and description | The named job and what it involves | Professional Barista Workflows |
| Skill level | Difficulty: Easy, Medium or Hard | Medium |
| Capture setup | How the task is recorded | Phone (head-mounted) / multiview rig |
| Status | How scoping would start | Active collection, Ready to deploy or Available on request |
13
Frequently asked questions
- What is a task taxonomy in robotics data?
- A structured hierarchy, such as sector, environment, task and skill, that organizes what a robot-training dataset covers so coverage, gaps and balance can be planned and audited.
- Is a task taxonomy the same as a robot task catalog?
- They overlap. A catalog is a taxonomy populated with concrete tasks and attributes such as description, skill level and status.
- How is a task taxonomy different from a skill spec?
- A taxonomy maps many tasks to plan coverage. A skill spec is the contract for one capture program, with geometry, condition hours, a reject rule and an acceptance test.
- How many levels should a taxonomy have?
- Programs differ. Sector, environment, task and skill is a common working structure; add a level only when it changes the data you need.
- Does being in Firsthand’s catalog mean a task has been captured?
- No. The catalog is a starting point for scoping. Capture runs against a signed spec, and unlisted tasks can be requested.
- Does a taxonomy replace hour targets?
- No. It shows where hours should go. A skill spec still budgets hours per condition for each task.
- Is a task taxonomy a file format?
- No, it is a planning artifact. Episodes ship in formats such as RLDS, LeRobot format, WebDataset or HDF5.
Related terms