Public vs Custom Robot Pretraining Datasets

Start with what a public corpus already covers. Ego4D and Egocentric-10K reach broad, large-scale daily-activity coverage that suits general pretraining; EgoDex adds strong hand-pose focus. If one of those already matches your task domain and its license permits your intended use, it is the faster and cheaper starting point.

A custom pretraining collection becomes the right call once a program needs a specific task or environment a public corpus does not cover, a commercial license a public dataset cannot grant, or synchronized streams — depth, IMU, 3D hand pose — a public corpus does not carry consistently. See /custom-collection/pretraining-data-collection for how a written spec becomes a delivered batch.

Ego4D is a large-scale, multi-site egocentric video corpus spanning broad daily activities, widely used as a general pretraining base. EgoDex focuses on egocentric video with an emphasis on hand pose, useful where dexterous hand data matters most. EgoVerse is a published research line studying egocentric video at scale for robot learning. Egocentric-10K, hosted on Hugging Face, released 10,000 hours of egocentric video under an Apache 2.0 license.

Each dataset differs mainly in scale, task/environment scope, label depth and license — not in whether it exists. See /compare/best-egocentric-video-datasets-for-robotics for what each is actually good for, and /compare/ego4d-vs-egodex for a direct comparison of the two most commonly paired corpora.

None of these releases is a substitute for checking its own documentation before planning around it. A public dataset’s scale and scope are fixed the moment it is released, so a program relying on one should confirm — directly from the dataset’s official page or paper — what tasks, environments and sensor streams it actually contains, rather than assuming coverage from its overall hour count alone.

Public dataset licenses vary and several of the largest are non-commercial: CC BY-NC and CC BY-NC-ND terms restrict commercial training and redistribution. Egocentric-10K is a notable exception, released under Apache 2.0, which permits commercial use under that license’s terms. Consent and provenance documentation also vary by dataset and is not always published per participant. See /compare/public-dataset-licenses for the full license comparison.

A custom collection is delivered on a perpetual, irrevocable, buyer-owned commercial license, with a written release per participant and a consent-plus-data-card artefact delivered per batch. That removes the license-diligence step a public corpus otherwise requires before every commercial use — see /guides/buyer-owned-training-data and /provenance.

A public dataset’s coverage is fixed at release — whatever tasks, environments and participants its original collection reached, and no more. That coverage is often broad (Ego4D, Egocentric-10K) or focused on one dimension (EgoDex’s hand-pose emphasis), but it cannot be extended to a gap a program discovers later without a new collection effort.

Custom collection puts coverage under the buyer’s control: tasks, environments and diversity targets are written into the spec before capture begins, so the resulting batch is built to close a known gap rather than reused from whatever a public release happened to capture.

Label depth and sync method vary by public dataset and are not documented as hardware-triggered on most public egocentric corpora — action labeling approaches range from manual annotation to automated estimation depending on the release. Programs pretraining on public data should check each corpus’s own documentation for what labels and sync guarantees actually ship with it.

A custom collection ships frame-accurate action segments, 3D hand pose and metric depth as standard synchronized streams, hardware-triggered on one clock. That consistency matters most when a program is mixing multiple data sources into one training run, since sync error compounds across sources that were not measured the same way.

A public dataset is available as soon as its license terms are confirmed to fit — no collection lead time, and typically free or covered by a one-time access agreement. That makes it the lower-cost, faster starting point whenever its existing coverage already fits the task.

A custom collection is priced per validated hour against a written spec, quoted before capture begins, and the timeline is set per engagement rather than being available on demand. The trade is calendar time and per-hour cost in exchange for coverage, sync guarantees and licensing built for exactly what the program needs — often started with a Spec Pilot, a small first batch that proves the spec before scaling volume.

The common pattern across programs is staged, not either/or: pretrain broadly on a public corpus like Ego4D or Egocentric-10K, then commission a custom batch to fill whatever the public data does not cover — a specific task, a specific environment, a commercial license, or synchronized depth/hand-pose streams the public corpus lacks. See /guides/robot-pretraining-data for how pretraining, mid-training and post-training divide that work across a full pipeline.

Firsthand delivers custom batches in RLDS, LeRobot, WebDataset, HDF5, zarr and Rerun .rrd, so a custom collection mixes cleanly alongside public-format data at whichever pipeline stage a program needs to fill.

Describe the task, environment and license gap a public corpus does not close. That becomes a written spec covering tasks, environments, capture tier, annotation depth and accept criteria, quoted before capture begins — see /custom-collection/pretraining-data-collection for the full engagement model.

Programs unsure whether the gap is real often start with a Spec Pilot — a small first batch collected against the written spec before committing to full volume. That lets a team validate coverage, sync and label quality on a limited scale before scaling a custom collection to replace or supplement what a public corpus already provides.