Glossary
What Is Multimodal Training Data?
Last reviewed: September 2026
Short answer
Multimodal training data is two or more aligned data streams — for example video, depth, audio, IMU, or pose — captured from the same event and time-synchronized so a model can learn the relationships between them, not just have them coexist in the same folder. For robotics and embodied AI, the streams that matter most are egocentric video, depth, IMU, hand or body pose, and the language or skill instruction that ties them to a task.
01
Why it matters
Streams that merely sit in the same folder but drift against each other, or that were captured at different times or from different embodiments, carry little of the cross-modal signal a model needs. Multimodal training data only pays off when the streams are hardware-synchronized on one clock and delivered with per-stream timestamps a loader can align.
02
How Firsthand handles it
Firsthand captures egocentric video, depth, IMU, and hand or body pose together, hardware-triggered onto a shared clock, to a written spec, on demand — delivered in RLDS, LeRobot format, WebDataset, HDF5, zarr, or Rerun .rrd with per-stream timestamps. See multimodal & sensor data collection for the full stream list and synchronization detail.
03
Streams that make up multimodal training data
| Stream | What it adds | Where it lives in a Firsthand episode |
|---|---|---|
| Egocentric video | The visual scene, from the embodiment’s own viewpoint | Head, wrist, or chest-mounted camera streams |
| Depth | Scene geometry aligned per frame | Synchronized depth stream, where the spec calls for stereo |
| IMU | Motion — accelerometer and gyroscope, hardware-timestamped on the same clock as video | IMU stream, hardware-triggered with the other sensors |
| Hand or body pose | 3D keypoints and contact events through the sequence | Pose stream, synchronized to the video and depth streams |
| Language or skill instruction | The task label that ties the streams to what the agent was doing | action_segment or per-episode task metadata |
05
Frequently asked questions
- What is multimodal training data?
- Two or more aligned data streams — for example video, depth, audio, IMU, or pose — captured from the same event and time-synchronized so a model can learn how they relate, not just find them in the same folder. See multimodal & sensor data collection for the full stream list.
- Why does synchronization matter for multimodal data?
- Streams that drift against each other, even slightly, break the cross-modal signal a model is meant to learn. Hardware-triggering every stream onto one clock is what keeps a dataset genuinely multimodal rather than several loosely related recordings.
- What streams count as multimodal training data for robotics?
- Egocentric video, depth, IMU, hand or body pose, and a language or skill instruction are the streams that matter most for robotics and embodied AI. See the table above for what each contributes and where it lives in an episode.
- Is multimodal training data the same as a world model or a VLA model?
- No. Multimodal training data describes the input — aligned streams captured together. A world model and a vision-language-action model are two different things that can be trained on it: a world model predicts how the scene changes, and a VLA model maps vision and language to an action.
- What formats is multimodal training data delivered in?
- RLDS, LeRobot format, WebDataset, HDF5, zarr, and Rerun .rrd, with per-stream timestamps and calibration included. See multimodal & sensor data collection for the full delivery detail.