Egocentric Pretraining Data for Robotics

Egocentric pretraining data is video recorded from the point of view of the person doing a task — a head, chest, or glasses-mounted camera showing hands, tools, and objects the way the actor sees them. It sits at the base of NVIDIA GR00T N1’s published data pyramid, where web data and human video occupy the widest, most abundant layer, beneath synthetic data in the middle and scarce real robot trajectories at the top.

What makes egocentric video specifically useful for pretraining, rather than just any first-person footage, is the viewpoint match: it shows a task from close to the angle a robot’s onboard camera would see it from, which is why it generalizes into a foundation model’s visual-motor priors more directly than third-person or web video does.

Egocentric human video scales in a way robot teleoperation data cannot: a person wearing a capture rig can move through many more tasks, rooms, and objects per hour than a teleoperated robot arm can, at a fraction of the hardware cost. That is the practical reason Generalist’s GEN-0 program describes accumulating 270,000 hours of real-world manipulation data and adding more than 10,000 new hours every week, and why other pretraining efforts converge on the same egocentric-human strategy rather than trying to scale robot-collected data directly.

Generalist’s own framing is a caveat worth keeping alongside that scale: "data quality and diversity matters more than sheer volume." Egocentric human video is the source that scales, but scale on its own is not the whole story — the diversity and label quality of that video is what determines how much a given amount of scale is actually worth.

NVIDIA’s EgoScale pretrained on 20,854 hours of action-labeled egocentric human video, with labels derived from off-the-shelf SLAM and hand-pose estimation, and reported a log-linear scaling law between that pretraining data’s scale and validation loss — a trend the paper states correlates with real-robot downstream task performance. Paired with a lightweight, aligned human-robot mid-training stage, that pretraining produced a 54% average success improvement over skipping pretraining entirely, measured on a 22-degree-of-freedom dexterous hand.

EgoVerse reports a smaller but still substantial egocentric corpus: roughly 1,362 hours across about 80,000 episodes. Read together with EgoScale’s tens-of-thousands-of-hours corpus and Egocentric-10K’s 10,000-hour, Apache 2.0-licensed dataset, the published egocentric pretraining corpora span a wide range of scale — the common thread across all of them is that the video is annotated or action-labeled, not raw unlabeled footage.

Monocular RGB video under-specifies a task: it shows what a scene looked like, not the 3D geometry, contact events, or precise hand motion a model needs to learn a manipulation prior from. A pretraining corpus built for that purpose adds synchronized depth for 3D geometry, 3D hand and body pose for motion and contact, and camera calibration so every stream lines up in the same physical space.

The way those signals are produced matters as much as whether they are present. EgoScale’s action labels come from off-the-shelf SLAM and hand-pose estimation applied after capture — a method that scales to tens of thousands of hours but estimates pose and contact rather than measuring them directly on a synchronized rig. Depth, hand pose, and action segments captured live, on the same hardware clock as the video, carry a cleaner signal per hour than labels reconstructed from monocular footage afterward.

Hand pose, contact events, and action segments are what turn raw egocentric footage into something a policy can learn from directly. This is the same distinction that separates imitation learning from a hand-designed reward: a policy trained by imitation learns from demonstrated trajectories, so its ceiling is set by demonstration quality — viewpoint, coverage, and label fidelity — not by an engineer’s intuition about what the reward should be.

A pretraining corpus feeding that kind of policy inherits the same ceiling. Action segments and hand pose are the signals an imitation-learning policy consumes directly, which is why they matter more to usefulness than an extra thousand hours of unlabeled video would.

A pretraining corpus is judged on breadth first: how many distinct skills, objects, and environments it spans, because that breadth is what the later, narrower mid-training and fine-tuning stages build on. Generalist’s own conclusion from GEN-0 — that data quality and diversity matter more than sheer volume — is a direct statement that hour count alone does not determine how useful a pretraining corpus is.

A custom egocentric collection plans that diversity explicitly through a written spec: which tasks, environments, regions, and demonstrators the corpus needs to cover, rather than capturing whatever footage is easiest to get in one location. That written-spec approach is what lets a program target the specific coverage gaps a model is missing, instead of hoping broad volume happens to include them.

Public egocentric corpora differ mainly in how they were captured and what they license. EgoScale’s labels come from off-the-shelf SLAM and hand-pose estimation rather than a purpose-built synchronized rig, and most large public egocentric datasets are silent on commercial licensing terms or ship research-only — Egocentric-10K, released under Apache 2.0, is the clear exception among the corpora referenced here.

A custom egocentric collection closes those gaps at once: hardware-synchronized depth and hand pose captured live rather than estimated afterward, coverage scoped to the tasks and environments a model is actually missing, and a license written for commercial training from the outset. This overlaps with, but is scoped more narrowly than, our broader pretraining data collection service, which spans the full pretraining corpus rather than the egocentric-video layer specifically.

The same capture stack used across every Firsthand program: ten synchronized streams per episode — 4K head camera, two wrist cameras, depth, IMU, 3D hand and body pose, instance segmentation, and action labels — hardware-aligned under 2 ms, so depth, pose, and video are fused signals rather than separately estimated ones.

Capture tiers scale from a phone in a chest mount, for the fastest path to broad geographic and household diversity, up to a head-mounted rig or a full synchronized multi-camera setup when a program needs the deeper 3D hand pose and contact signal manipulation-focused pretraining depends on. Faces, plates, and screens are detected and irreversibly blurred at ingest, confirmed by a reviewer before anything ships.

Start with the coverage gap: which tasks, objects, environments, or demonstrator diversity your current pretraining corpus is missing. That becomes a written spec covering capture tier, annotation depth, and accept criteria, quoted for reach, timeline, and price before capture begins.

Many programs start with a Spec Pilot ��� a small, fast first batch that proves the spec and label quality before scaling volume ��� then run through the same seven-stage pipeline used for every Firsthand collection, with a median 48-hour first validated batch where coverage is deep, and delivery in RLDS, LeRobot, WebDataset, or HDF5 under a buyer-owned perpetual license.