Guide
Egocentric Video Data for Embodied AI
Short answer
Egocentric video is first-person footage — camera at eye or chest height — paired with synchronized depth, pose, and action data. Embodied AI models need it because it captures how humans actually see and manipulate the world, which third-person video and simulation don’t. It’s available from public research datasets or through custom, on-demand collection built to a spec.
Last reviewed:
What is egocentric video data?
Egocentric video is captured from a first-person viewpoint — a camera worn on the head or chest, or a phone held at chest height — instead of a fixed third-person angle. On its own it is one video stream. Paired with synchronized depth, IMU, and 3D hand or body pose on a common clock, it becomes training data an embodied AI model can learn from: what the hands are doing, where the camera is relative to the scene, and how both change over time.
- Viewpoint
- First-person — camera at eye or chest height, moving with the body.
- Companion streams
- Depth, IMU, and 3D hand or body pose on the same clock as the video.
- Why it matters for training
- Captures the hand-eye relationship a fixed third-person camera or simulation cannot.
- Delivery formats
- RLDS, LeRobot, and WebDataset — the formats robot-learning pipelines already read.
Why does embodied AI need egocentric video specifically?
Embodied AI models — for manipulation, humanoid control, or world modeling — need to learn how a body interacts with a physical scene. A first-person view puts the camera where the acting body’s eyes or chest would be, so the video already encodes the viewpoint the model has to reason from at inference time. Third-person footage shows the task from the outside; egocentric footage shows it from the perspective that actually has to act on it.
Where does egocentric video data come from?
| Source | What you get | Best for |
|---|---|---|
| Public research datasets | Fixed scope, fixed license, no control over environments or tasks. | Prototyping and benchmarking against a known baseline. |
| Custom, on-demand collection | A spec you define — environments, tasks, geography, streams, formats. | Filling a coverage gap or shipping a licensed dataset for commercial training. |
Public datasets and custom collection solve different problems — most teams end up using both.
Public datasets are the fastest way to get started, but their scope, license, and coverage are fixed by whoever released them. Custom collection is slower to set up but built to your spec, with a buyer-owned license and full consent documentation, and can ship a first validated batch on a median of 48 hours once the spec is locked.
How fast can egocentric video data ship?
For custom collection, turnaround depends on how common the coverage cell is. A deep, common cell (a typical kitchen task, for example) can ship a first validated batch in a median of 48 hours; a thin cell — an unusual environment, condition, or task combination — takes longer because it has to be sourced, not just pulled from an existing pool.
FAQ
Egocentric Video Data for Embodied AI, answered.
01Is egocentric video the same as first-person video?
Yes — the terms describe the same thing: footage captured from the viewpoint of the person or body performing the task, rather than from an external, third-person angle.
02Do I need depth and pose data, or is video enough?
Video alone can support some computer-vision tasks, but for manipulation, humanoid control, or world models, most training pipelines need synchronized depth, IMU, and hand or body pose alongside the video — not video in isolation.
03Can I mix public datasets with custom-collected data?
Yes. Many teams prototype against a public dataset, then commission custom collection to cover the environments, tasks, or geographies the public data is missing, and combine the two in training.
04What formats does egocentric training data come in?
Custom-collected data from Firsthand is delivered in RLDS, LeRobot, or WebDataset — the same formats most robot-learning and imitation-learning pipelines already read.
05Who owns the license to custom-collected egocentric video?
For custom collection, the buyer owns the resulting dataset outright, under a perpetual license, with a documented consent chain for every participant.
Sources
Where is this documented?
- RLDS (GitHub) (opens in a new tab)Reference format for reinforcement- and imitation-learning episode data. github.com
- LeRobot (Hugging Face) (opens in a new tab)Open robot-learning dataset format and tooling. github.com
Check it against the sample pack.
40 episodes across 4 environments, delivered in the exact schema these guides describe.