Pretraining Data for Robot World Models

A world model learns to predict how a scene changes in response to an action — the next frame, state, or few seconds of video given what an agent just did. During pretraining, that prediction target is learned broadly, across many tasks, scenes, and objects, so the model builds a general predictive prior before any downstream policy or fine-tuning stage narrows it. That takes dense egocentric video paired with the action, pose, or depth signal that caused each state change, captured across enough diversity that the model does not overfit to one narrow setting.

This differs from policy pretraining data, which is judged on action-label quality for a control target. World-model pretraining data is judged on causal density — how reliably each frame is paired with the signal that explains why the scene changed — and on how many distinct conditions that pairing is captured across.

Egocentric video puts the camera at the embodiment doing the acting — a head or chest mount, or a wrist camera on the hand itself — so the viewpoint moves with the action instead of watching it from outside. That matches the viewpoint a robot or embodied agent will actually have at inference time, and it keeps hands, tools, and contact points in frame at the moment they matter, rather than off to the side of a fixed third-person shot. See egocentric video for world models for the full breakdown of why viewpoint transfer matters for this prediction target.

At pretraining scale, that viewpoint advantage compounds: a corpus built from head, wrist, and chest mounts across many tasks and environments gives a world model the same first-person geometry it will act through at deployment, across far more conditions than a fixed-camera dataset could practically cover.

Video alone only teaches a model to predict pixels. To predict the effect of an action, the training data has to carry the action itself on the same timeline as the frames — stereo depth grounds the predicted change in scene geometry rather than pixel change alone, and IMU or hand-pose data supplies the motion signal that explains why the scene changed. All of it needs to share one hardware clock so a given frame and a given cause are provably simultaneous rather than merely close in time.

Every sensor in the rig hardware-triggers onto one PTP clock domain, so video, depth, IMU, and hand pose share a timebase rather than being merely co-recorded — the same synchronization requirement described in egocentric video for world models. That sync is what makes a frame and the cause inside it usable as a training pair, at pretraining scale rather than in a handful of clean clips.

A raw video stream shows what happened, but a world model needs to learn what caused it. Action labels — 3D hand pose, contact events, or robot action logs — attached to the same synchronized clock as the video are what let a model condition its prediction on a specific action rather than merely on the passage of time. Without that pairing, a model can learn scene dynamics in general but cannot learn how a specific action changes a specific scene, which is the target most world-model training actually needs.

The same action-label families used across Firsthand’s action-labeled egocentric video pages apply here: hand pose, contact/grasp events, and action segments, captured live against synchronized depth rather than estimated from raw footage after the fact.

A world-model pretraining corpus is planned for coverage across environment, lighting, and task condition cells, rather than one location or one actor, so the model learns to predict state changes across the range of conditions it will encounter downstream. That is the same diversity-first planning approach described for robot pretraining data generally, applied specifically to the video-and-cause pairing a world model needs.

Generalist’s GEN-0 program frames this directly: "data quality and diversity matters more than sheer volume." Scale is a means to reach that diversity, not the end goal — a corpus judged only on hour count can still be narrow in the conditions it actually covers.

In practice, a written spec sets that diversity budget explicitly — how many environments, how many lighting conditions, how many distinct actors or objects per task cell — rather than leaving coverage to whatever a fixed capture location happens to produce. Coverage is validated cell by cell before delivery, the same accept-criteria discipline used across every Firsthand pretraining collection, so a program can confirm the spread it asked for actually landed in the delivered corpus rather than discovering gaps once training has already started.

A vision-language-action or imitation-learning policy pretrains to map an observation (and often an instruction) directly to an action — its training signal is the action itself. A world model pretrains to predict the next state given an action, without necessarily outputting a control command at all. Both need dense egocentric video and a synchronized action or pose stream, but a policy dataset is judged on how well the action label matches a controllable output, while a world-model dataset is judged on how reliably the video-action pairing lets the model predict forward.

In practice, the same episode can often serve both purposes: egocentric video, hand pose or action logs, and (for a VLA target) a language instruction can all be captured on one synchronized clock in the same episode, so one collection can support multiple downstream training targets. See what is a world model? and what is a vision-language-action model? for how the two targets differ.

Public video corpora were captured for someone else’s purpose, so the action or pose signal a world model needs may be missing, approximate, or licensed for research only. A custom collection is scoped to the exact tasks, environments, and signal fidelity a world-model program needs — hardware-synchronized video, depth, and action signal, captured to a written spec, under a license written for commercial training from the outset.

Many programs mix both: a public corpus for early experimentation, then a custom collection scoped to fill the specific gaps — conditions, viewpoints, or action vocabularies — the public data does not cover.

Start with the task and environment diversity a world model needs to predict across, and whether the spec calls for stereo depth or monocular video alongside the action or pose signal. That becomes a written spec, quoted for reach, timeline, and price before capture begins.

Many programs start with a Spec Pilot — a small, fast first batch that proves coverage and sync quality before scaling volume — then run through the same seven-stage pipeline used for every Firsthand collection, with a median 48-hour first validated batch where coverage is deep, delivered in RLDS, LeRobot, or WebDataset under a buyer-owned perpetual license.