Egocentric Video Data for World Models

A world model learns to predict how a scene changes in response to an action — the next frame, the next state, or the next few seconds of a video given what an agent just did. That prediction target only exists if the training data pairs dense visual frames with the action, pose, or contact event that caused the change. Static image sets or unpaired video clips have no causal signal to learn from.

Egocentric video puts the camera at the embodiment doing the acting — a head or chest mount, or a wrist camera on the hand itself — so the viewpoint moves with the action instead of watching it from outside. That matches the viewpoint a robot or embodied agent will actually have at inference time, and it keeps hands, tools, and contact points in frame at the moment they matter, rather than off to the side of a fixed third-person shot.

• The viewpoint transfers: a policy trained on a third-person view has to relearn geometry from a first-person camera at deployment
• Hands and contact stay in frame at the moment of action, rather than occluded by the actor’s own body from outside
• Head, wrist, and chest mounts can all run in the same episode on one synchronized clock

Video alone only teaches a model to predict pixels. To predict the effect of an action, the training data has to carry the action itself on the same timeline as the frames — 3D hand pose, IMU motion, depth, or robot action logs, all hardware-triggered onto one clock so a given frame and a given action are provably simultaneous rather than merely close in time.

Every sensor in the rig hardware-triggers onto one PTP clock domain, so video, depth, IMU, and hand pose share a timebase rather than being merely co-recorded — the measured residual across delivered episodes runs a median of 1.12 ms and a p99 of 1.94 ms, with a 2.0 ms reject ceiling. That sync is what makes a frame and the action inside it usable as a training pair.

The full sync measurement, verified against real episode data, is in "How to measure cross-stream sync error" below.

A world-model dataset is collected against a written spec that names the task, the environments, and the condition budget, rather than pulled from a fixed catalog — so reach, timeline, and price are quoted against what the model actually needs. A first validated batch ships in a median of 48 hours where coverage is deep.