Diverse Egocentric Video Data Collection

Diverse egocentric video data is first-person footage that covers a deliberately planned spread of people, places, tasks, objects and conditions, rather than many hours of the same few. The key word is planned. Diversity that is hoped for tends to collapse toward whatever is easiest to film, so on this site it is written into the skill spec as targets before capture begins.

Diversity is not the same thing as volume. A dataset can accumulate many hours while still drawing on a narrow set of demonstrators, rooms or tasks. Generalist’s GEN-0 write-up states the same principle in its own words: data quality and diversity matters more than sheer volume. Coverage is what turns that principle into something you can check before training starts.

It is also separate from the other properties buyers ask about. Consent covers the people in the footage, the license covers what you may do with it, and diversity covers what the footage spans. This page is about the last of those and how a custom collection plans for it.

A custom collection can plan for six axes, each tracked independently. Contributor is the variation across the people performing the task: hands, grip technique, pace, movement style and handedness. Scene is the number of distinct rooms, layouts and backgrounds, not hours in each. Task is the exact task verb and noun rather than a small set repeated. Object is distinct categories and instances. Condition is lighting, clutter and setting. Failure case is dropped objects, occlusion, low light and recovery, not only clean attempts.

The axes combine into condition cells. A given scene, task and lighting condition together form one cell, and the spec budgets hours per cell with its own minimum target. That turns an abstract goal into a grid you can review cell by cell before capture starts, and it is why a corpus can look on pace by total hours while several cells sit near zero.

Participant diversity here is a planning and QA concern, not a demographic report. It is tracked through coverage targets and a contributor identifier tied to the consent record on each clip. The first table below lists each axis and how it is recorded.

We design the first draft of the skill spec with you on a call, including hours budgeted per condition cell. The spec names the scenes, task variations, object classes and contributor mix a batch must cover, and states a minimum share for each. The vocabulary and the coverage requirement live in one document, so labels and diversity checks refer to the same source of truth.

There is no universal number that fits every program. A narrow single-task corpus needs less environmental spread than a general-purpose corpus meant to transfer across many downstream tasks, so how many contributors, scenes, regions and objects a program covers is set per program in the skill spec rather than assumed here.

Capture tier is part of the balance. The phone tier, a contributor filming hands-free with a phone in a chest mount, is the fastest way to reach broad geographic and household diversity. The head-rig tier adds IMU data for tasks where gaze and head motion matter, and the multi-camera tier synchronizes head, wrist and depth streams for manipulation work that needs 3D hand pose and contact information. Programs often mix tiers under one spec.

A policy learns the statistical pattern of what it was shown. If every clip comes from one room, one lighting setup and one person, the model has no evidence the task looks different anywhere else, so the first new environment is where it has least to generalize from. That is the practical reason a pretraining plan treats diversity as a primary signal alongside hours.

Published work points the same way, with caveats. The EgoVerse paper reports that increasing demonstrator diversity improves robustness to unseen human embodiments, while scene diversity plays a dominant role in generalization to novel environments, particularly under limited data budgets. Its authors also note that the diversity experiments rely on offline metrics and that additional robot rollouts are needed to confirm the effect on robot manipulation.

We state no result for our own data. What we can say is narrower: a written target per axis lets you check a batch against the plan, which is the part a buyer controls before training begins.

Coverage is tracked per batch during collection, not only once at delivery. Fill rate against each cell is reported so both sides look at the same picture a batch is accepted against. The check is on distribution as well as count: if one scene, task variation, object class or contributor accounts for a disproportionate share of hours, that axis is under-diversified even when the number of distinct values looks reasonable.

The pilot is where the plan meets real footage. A Spec Pilot is reviewed against every axis in the spec before volume scales, with the same reject log and validated-hour billing a full collection gets. A pilot that is on pace by hours but concentrated in a few easy cells is a signal to adjust the plan, not to proceed.

Failure cases need their own budget. Without one, the share of clean successes tends to grow as volume grows, so failure and recovery are budgeted explicitly as a share of each condition cell rather than left to chance. Rejected footage is logged with a reason and visible to you, so you can see what was discarded and why.

Public datasets document diversity in their own ways, so the fair comparison is attribute by attribute, using each project’s own published description. The third table below does that for EgoVerse and Egocentric-10K, citing only what their official pages state. Where a page does not state something, the table says so rather than guessing.

The structural difference is control. A public dataset’s diversity has to be assessed from its documentation after the fact, while a custom collection lets you set and check coverage targets directly before and during capture. A public release can still be the right fit for exploration or a baseline. The two are not exclusive, and many teams use both.

License and consent are the other things to check on any public source, because they differ by dataset. Ours are stated on this site: consent is documented per episode and the license is perpetual, irrevocable and buyer-owned.

Diversity is only checkable if each episode carries the facts to count it. Scene and location metadata, a task label with action-segment labels, object class labels set by the skill spec before capture, condition metadata for lighting and setting, and an outcome flag with recovery action-segment labels sit alongside a contributor identifier tied to the consent record on each clip.

The fourth table below lists what each records and which diversity question it answers. It describes the information by what it records. The exact field names and layout follow the delivered episode schema, and which of these a given program attaches is set per program in the skill spec.

Each batch also ships with a data card and a reject log, so the same batch you count coverage on is the batch you check provenance on.

Describe the diversity your corpus is missing: which environments, tasks, objects and kinds of contributor, and which failure cases. That becomes a written spec with a minimum target per axis, quoted for reach, timeline and price before any capture begins.

Many programs start with a Spec Pilot to prove the targets hold up on real footage. Where coverage for your tasks and regions is deep, the median time from signed spec to first validated batch is a median of 48 hours; where it is thin, three to four weeks while recruiting expands. Anything this page does not state, such as contributor, scene or region counts, is set per program in the skill spec.