How to Plan Diversity in a Robot Pretraining Dataset

A pretraining corpus is judged on how well it teaches general visual, physical and action priors before a model ever sees its target robot or task. A large hour count does not guarantee that: footage can be voluminous and still statistically narrow if it comes from a small number of scenes, a small set of tasks, or a small pool of contributors repeated at scale. The robot pretraining data quality checklist makes this explicit — diversity across scenes, tasks, objects and contributors is checked as a primary signal, independent of total hours.

GEN-0’s own public framing backs this directly: the team states that data quality and diversity matters more than sheer volume. That is consistent with why dataset diversity matters in robot learning — a policy only generalizes to conditions it has actually seen, so a dataset filmed in one room under one lighting setup with one demonstrator teaches a model that narrow slice, not the underlying task.

The practical implication is that diversity cannot be left to chance or assembled after the fact from whatever footage happens to exist. It has to be planned into the collection spec as explicit coverage targets, the same way a capture spec names the task, the labeling method and the acceptance test. This guide walks through how to turn diversity goals into targets that can be written into a spec, tracked during collection, and verified before a corpus is scaled up.

A pretraining diversity plan is built around a small number of axes that each need to vary independently. No single axis compensates for a gap in another: a corpus with many scenes but one task type is still narrow, just along a different dimension than a corpus with many tasks filmed in one room.

These axes are the same ones used throughout the robot pretraining data quality checklist and why dataset diversity matters in robot learning — this guide focuses specifically on turning them into coverage targets before capture, rather than auditing them after delivery.

A diversity goal only becomes verifiable once it is written into the skill spec as a minimum target, in the same way a capture spec names the acceptance test for a task. Instead of a general instruction to "capture varied footage," the spec names the specific scenes, task variations, object classes and contributor mix a batch needs to cover, and states a minimum share of hours or episodes required for each.

This mirrors how failure-case coverage is already specified: a spec that defines what counts as a failure case and sets a minimum share of total hours for that category is verifiable after delivery in a way that an unquantified claim is not. The same approach applies to every diversity axis — a target stated as a minimum number of distinct scenes, task variations, object classes or contributors gives both the collection team and the buyer a concrete number to check the finished corpus against.

Because the skill spec is what defines the object classes, grasp families and environments in scope before capture, it is also the natural place to define diversity targets — the vocabulary and the coverage requirement live in the same document, so labels and diversity checks both reference the same spec rather than two separate sources of truth.

One practical way to plan diversity is to treat the combination of axes as a grid rather than a list: a given scene, task and lighting condition together form a single cell, and the collection target becomes a minimum number of episodes in each cell rather than a single aggregate hour count. This turns an abstract diversity goal into a concrete planning tool — a program mix can be reviewed cell by cell to see where coverage is thin before capture starts, not after a corpus is delivered.

Tracking how full each cell is against its target — its fill rate — during collection gives an early warning that is not visible from a running hour total alone. A corpus can be on pace by total hours while several cells sit near zero, because the easiest scenes, tasks or lighting conditions to capture tend to fill first unless the collection plan actively directs effort toward the cells that lag behind.

This is the same representative, to-spec program mix described in why dataset diversity matters in robot learning, made operational: the environments, lighting conditions and participant mix a batch needs to cover are named in the spec up front, and progress against each is tracked explicitly rather than assumed from aggregate volume.

There is no universal number that applies to every pretraining program, because the right amount of diversity depends on the skill spec’s scope — a narrow single-task pretraining corpus needs less environmental spread than a general-purpose corpus meant to transfer across many downstream tasks. The useful question is not "how many is enough" in the abstract, but whether the corpus’s diversity matches the breadth of behavior the resulting model is expected to generalize to.

A qualitative check that applies regardless of scale: if any single scene, task variation, object class or contributor accounts for a disproportionate share of total hours, that axis is under-diversified relative to the rest of the corpus, even if the absolute count of scenes or contributors looks reasonable on its own. Reviewing the distribution across each axis — not just the count of distinct values — is what catches a corpus that technically spans many scenes but is still dominated by one or two of them.

This is also where a pilot becomes useful: rather than guessing at target counts up front, a small pilot sample can be reviewed for how evenly it spans the planned axes, and the targets adjusted before the full collection scales up.

Failure-case coverage is one of the six diversity axes above, but it deserves separate attention because it behaves differently from the others: failure and near-miss cases require deliberate planning to capture, and as collection volume grows without that planning, the percentage of a corpus made up of clean successes tends to increase on its own. A dataset built entirely from successful demonstrations does not teach a model what a stall, slip, or correction looks like, which matters for the same reason environmental diversity matters — the model has no evidence of what it was never shown.

The fix is the same one used for every other diversity axis: define what counts as a failure case (a dropped object, a missed grasp, a recovery attempt) in the collection spec, and set a minimum share of total hours or episodes for that category. See how to capture failure cases in robot training data for the detail on how failure and recovery behavior is specified, captured and labeled during a collection.

A pilot sample is the point at which a diversity plan is tested against real footage rather than a written target. Checking a pilot for diversity means comparing the delivered sample against each axis in the spec: did the pilot span the planned scenes, task variations, object classes and contributors, or did it concentrate in the easiest cells to capture? A pilot that is on pace by hour count but concentrated in a handful of cells is a signal to adjust the collection plan before it scales, not a reason to proceed.

This is the same spec-first, pilot-before-scale approach described in how to collect robot pretraining data: write the spec, run a pilot, review it against the spec, and only then scale collection. Checking diversity at the pilot stage is far cheaper than discovering a coverage gap after a full corpus has already been delivered.

Public pretraining corpora document their scale differently, which makes direct diversity comparisons difficult without relying on each project’s own published description. EgoVerse reports roughly 1,362 hours across approximately 80,000 episodes, a structure that implies many short, distinct episodes rather than a small number of long continuous recordings. Egocentric-10K is released as 10,000 hours of egocentric video under an Apache 2.0 license on Hugging Face, with its own documentation describing collection scope. Figure has published a description of its large-scale data collection effort, Project Go-Big, on its own site.

None of these public descriptions substitute for reviewing a corpus’s actual diversity before using it, for the same reason a large hour count does not guarantee diversity on its own: the right question for any corpus, public or custom, is how its hours are distributed across scenes, tasks, objects and contributors, not just how many hours are reported in total.

An end-to-end diversity plan for a custom pretraining collection follows the same sequence as the rest of the collection spec: name the diversity axes relevant to the target model (scene, task, object, contributor, condition, failure case), set a minimum coverage target for each in the skill spec, run a pilot, review the pilot against every axis before scaling, and carry the same targets through the full collection with fill rates tracked along the way rather than only checked at the end.

Writing the plan this way keeps diversity verifiable at delivery rather than something a buyer has to take on trust: the coverage target for each axis is stated up front, checked in the pilot, and can be checked again against the delivered corpus, in the same way the acceptance test, labeling method and failure-case share are each checked against the written spec rather than a general quality claim.