Pretraining Data for Robotics Startups

A robotics startup building or fine-tuning a robot foundation model needs the same base-layer data any foundation model needs: broad, first-person coverage of human behavior across tasks, objects, and environments, captured with manipulation-grade signal — hand pose, depth, and action labels — rather than raw RGB alone.

What differs for a startup is scale and commitment. A written spec can be sized to an early program’s actual roadmap — a focused set of tasks and environments — rather than requiring the budget or volume a larger lab would commit to on day one.

Public egocentric corpora are a legitimate way to build broad, general priors — EgoScale’s own results show a research-grade corpus, action-labeled by off-the-shelf SLAM and hand-pose estimation, is enough to demonstrate a real scaling law and a measurable success-rate gain. The limitation for a startup is coverage and licensing, not whether scale is achievable at all.

A public corpus was captured for someone else’s task distribution, so it is always missing exactly what a given model needs next, and most large egocentric corpora carry non-commercial or research-only terms rather than a right to train a commercial product on them. A custom collection closes both gaps: coverage scoped to the exact tasks and environments a startup’s model is missing, and a license written for commercial training from the outset.

Per NVIDIA’s GR00T N1 data pyramid, pretraining sits at the base: web data and human video, because they are the most abundant. Mid-training sits above that with more targeted, aligned data, and real robot trajectories sit at the top because they are scarcest and most expensive. A pretraining-stage purchase is judged on breadth across tasks, objects, and environments — not on how closely it matches a single target robot embodiment.

A startup working through all three stages does not need a separate data provider or ingestion pipeline for each: the same ten-stream capture stack and delivery formats (RLDS, LeRobot, WebDataset, HDF5, zarr, Rerun .rrd) carry across pretraining, mid-training, and fine-tuning data.

Start from the gap in your model’s current coverage rather than a target hour count: which tasks, objects, or environments is the model still weak on. That becomes a written spec covering tasks, environments, regions, capture tier, annotation depth, and accept criteria, quoted for reach, timeline, and price before any capture begins.

Because the spec drives both coverage and cost, a startup can start narrow — a handful of tasks and environments — and widen the spec once the pilot has proven out quality and format fit, rather than committing to broad coverage before knowing whether the pipeline integration works.

Generalist AI’s GEN-0 program, at 270,000+ hours and growing by 10,000+ hours per week, states explicitly that data quality and diversity matter more than sheer volume. For a startup with a limited budget, that means a smaller, more diverse, well-labeled corpus scoped to the model’s actual weak spots is a better pretraining investment than maximizing hour count alone.

Every episode in a custom collection ships with hand pose, depth, and action labels captured live on the rig rather than estimated from raw video afterward, so the manipulation-grade signal that generalization research points to is present from the first batch, not added later.

Ten synchronized streams per episode: 4K head camera, two wrist cameras, depth, IMU, 3D hand and body pose, instance segmentation, and action labels, hardware-aligned under 2 ms. Nothing in this stack is estimated after capture — pose and action labels come from the rig and the capture pipeline, not a post-hoc annotation pass on raw RGB.

Coverage (tasks, environments, regions) is set by the written spec a startup provides, not a fixed catalog of available footage, so the corpus fills the specific gap the model needs rather than being generally useful to everyone.

Every episode links to a signed contributor release through a documented consent chain. Faces, plates, and screens are detected and irreversibly blurred at ingest, and a reviewer confirms the result before anything ships, with the anonymization record delivered alongside the data.

Delivered data carries a perpetual, buyer-owned commercial license, with exclusivity available so the same data is never resold to a competitor — the licensing clarity many public pretraining corpora do not provide for commercial use.

Many programs, including early-stage teams, start with a Spec Pilot: a small, fast first batch that proves the spec and accept criteria before committing to scale. It lets a startup check stream quality, hand-pose and action-label fidelity, and format compatibility with its own training pipeline before volume ramps.

From there, the purchase scales through the same seven-stage pipeline used for every Firsthand program — contributor recruiting, capture, per-batch QA, anonymization, and licensed delivery — with rejected items recaptured at Firsthand’s cost. A median 48-hour turnaround on a first batch means a startup can validate the spec quickly, then scale to the volume its model needs, with on-demand top-ups available afterward as new gaps show up during training.