Glossary
What Is a World Model?
Last reviewed: September 2026
Short answer
A world model learns to predict how a scene changes in response to an action — the next frame, the next state, or the next few seconds of video given what an agent just did. That prediction target only exists in training data when dense visual frames are paired with the action, pose, or contact event that caused the change; static image sets or unpaired video clips carry no causal signal to learn from.
01
Why it matters
A world model is only as good as the causal signal in its training data — it needs video dense enough to show the state change, plus the action or pose data that explains why the scene changed. Video without an aligned cause, or actions without dense enough video to show their effect, both fail to teach prediction.
02
How Firsthand handles it
Firsthand captures dense egocentric video alongside depth, IMU, and hand or action signals, hardware-synchronized onto one clock, to a written spec, on demand — the paired observation-and-cause a world model needs to learn from.
03
What a world model learns from each stream
| Stream | What it contributes to the prediction target | Where that lives in a Firsthand episode |
|---|---|---|
| Egocentric video | The visual state to predict forward from and to | Head, wrist, or chest-mounted camera streams |
| Depth | Scene geometry, so predicted change is spatial, not just pixel change | Synchronized depth stream, where the spec calls for stereo |
| Hand pose or action stream | The cause paired with the visual effect — what the agent did to change the scene | action_segment fields, hardware-synchronized to the video streams |
05
Frequently asked questions
- What is a world model in AI?
- A world model learns to predict how a scene changes in response to an action — the next frame, state, or few seconds of video given what an agent just did. See egocentric video for world models for the full breakdown of what training data it needs.
- What data does a world model need to train on?
- Dense visual frames paired with the action, pose, or contact event that caused the change — static image sets or unpaired video clips carry no causal signal for the model to learn from.
- Does a world model need stereo depth?
- Depth is not strictly required, but it grounds the predicted change in scene geometry rather than pixel change alone. See egocentric video for world models for when a spec calls for stereo depth.
- Is a world model the same as a vision-language-action model?
- No. A vision-language-action (VLA) model maps a visual scene and a language instruction to an action; a world model predicts how the scene itself changes in response to an action, with no instruction or language input required. The two can be trained on overlapping episodes, but they learn different targets.
- Can the same dataset train both a world model and a behavioral-cloning policy?
- Yes, if the episode has the same aligned observation-action structure both methods need — dense video plus the action or pose stream that caused each change, on one synchronized clock.