Who Collects Robot Pretraining Data?

Robot pretraining data — the large-scale, general-purpose layer of video and action labels a robot foundation model learns from before it is aligned to any specific robot or task — comes from four distinct categories of source, not one. AI labs and research groups collect some of it in-house. Some of it is released publicly as research corpora. Some is captured by outside data collection partners hired specifically for scale. And some is captured on demand by a custom collection service, against a written spec, for a program that needs coverage no existing source provides.

These categories are not mutually exclusive. A single pretraining program commonly blends two or three of them at once: a public corpus for early, broad coverage; in-house or partner-collected data for scale; and a custom collection engagement to fill the specific gap the blended mixture is still missing. The rest of this guide walks through each category, what distinguishes it, and how the four compare when weighed against each other.

Yes, some frontier AI labs run their own in-house data collection operations rather than relying only on external sources. Generalist AI’s GEN-0 program is a public example: the company describes accumulating 270,000 hours of real-world manipulation data and adding more than 10,000 new hours every week, and states directly that "data quality and diversity matters more than sheer volume." Figure’s Project Go-Big is a comparable example of a robotics company describing its own large-scale data collection effort in similar terms on its official announcement.

In-house collection gives a lab direct control over task selection, environment coverage, and quality bar, and it lets the collection effort run continuously alongside model development rather than as a separate procurement step. The tradeoff is that building and operating a capture program — recruiting demonstrators, maintaining hardware, running quality control — is itself a significant operational undertaking, distinct from the model-training work it is meant to support.

A second category is data released publicly by research labs, universities, and AI companies specifically as pretraining corpora for the broader robotics research community. These differ in who built them, how large they are, and what license they carry.

Public corpora are a fast way to start: the data already exists, and several of the sources above document real scaling results, including EgoScale’s log-linear scaling law between pretraining data volume and validation loss. The tradeoff is fixed scope — the tasks, environments, and object sets were chosen by whoever released the corpus, not by the team using it — and license terms that vary from source to source, with some academic egocentric datasets shipping under research-only terms that do not clear commercial training.

A third category sits between fully in-house collection and a public corpus: hiring an outside partner to run collection at scale, either as a standing program or as a large one-off capture effort. Labs reach for this when they need more hours or more coverage than their own in-house team can reasonably produce, but still need the result scoped to their own tasks and environments rather than a fixed public release.

The evaluation questions for this category are the same ones that apply to any robotics data vendor: what sync error, annotation depth, license terms, and consent documentation does the partner actually deliver, and can they sustain that quality at pretraining scale — across many scenes, tasks, objects, and demonstrators — rather than only on a small, convenient sample. See how to choose a robot pretraining data provider for the full buyer checklist.

A custom pretraining data collection service is a specific kind of outside partner: rather than a standing collection program, it runs on demand, against a written skill spec, and is scoped narrowly to the exact gap a given pretraining corpus is missing. Firsthand runs this model for the pretraining layer specifically: a written spec names the tasks, environments, and coverage a program needs, a small pilot batch validates that spec before any volume commitment, and full-scale capture follows the same seven-stage collection pipeline used across all custom collection work.

What is captured is the same stack used across every Firsthand program: ten synchronized streams per episode — 4K head camera, two wrist cameras, depth, IMU, 3D hand and body pose, instance segmentation, and action labels — hardware-aligned under 2 ms. Because hand pose, depth, and action labels are captured live on the rig rather than estimated from raw video after the fact, every episode carries the same manipulation-grade signal at pretraining scale that a smaller fine-tuning batch would carry. Where coverage is already deep, a first validated batch ships in a median of 48 hours; thin or specialized environments typically take three to four weeks.

A custom pretraining data collection service does not replace public corpora or in-house collection — it fills the specific gap those sources leave: a task, environment, or camera setup a program needs that nothing already captured provides, delivered under a license written for commercial training from the outset.

Each category trades off differently on the properties that actually matter for pretraining: how much data is available, whether it ships with action labels and calibration, what the license permits, and how fast a team can get a usable batch.

Licensing is worth calling out separately, because it is where the categories differ most. Public corpora carry whatever license the releasing organization chose, and several academic egocentric datasets ship under research-only terms that do not automatically permit training a model that will later ship commercially. In-house and custom-collected data avoid that ambiguity by design, since the collecting organization sets the license from the start — but an outside partner’s license still needs the same confirmation a buyer would apply to any vendor.

The choice is rarely one source versus another — it is which combination fits a given program’s stage and gap. A program just getting started can lean on a public corpus for broad, general coverage at effectively no collection cost. A program with an established in-house capability and a task distribution that matches its own operational reach can build that capability out further. A program that needs more volume than it can produce alone, scoped to its own tasks, is a fit for an outside collection partner. And a program with a specific, known gap — a task, environment, or camera setup nothing already captured provides — is a fit for a custom collection service run against a written spec.

Whichever source (or combination) a program chooses, the same evaluation questions apply: what is the measured sync error, what action labels and calibration ship with the data, what license governs commercial use, and how fast can a first validated sample actually be produced. Start with a small sample or Spec Pilot against your own spec before committing to volume from any source — see how to evaluate a robotics data vendor for the full due-diligence process.