Glossary
What Is Egocentric Data Collection?
Last reviewed: October 2026
Short answer
Egocentric data collection is the process of capturing first-person video — plus the depth, pose, and action streams recorded alongside it — from a camera worn by the person performing a task, so the resulting data shares the viewpoint a robot’s own head or chest camera would see. It is the activity that produces egocentric video, not the footage itself.
01
Why it matters
A robot foundation model trained on third-person or internet video learns a viewpoint it will never occupy at inference time. Egocentric data collection exists to close that gap by recording from the actor’s own point of view, with the calibration, synchronization, and labels needed to make the result trainable rather than merely watchable — which is what separates egocentric data collection from simply filming someone with a head-mounted camera.
02
How Firsthand handles it
Firsthand runs egocentric data collection as a written-spec, on-demand service: a head-mounted global-shutter camera plus two wrist cameras, all hardware-triggered onto one PTP clock domain, capturing synchronized RGB, depth, 3D hand pose, and action-segment labels. Consent is tracked through a signed consent chain, faces and identifying details outside the task are anonymized, and the result is delivered in RLDS, LeRobot format, or WebDataset with a buyer-owned commercial license.
03
What an egocentric data collection captures, stream by stream
| Stream | What it captures | What it’s for |
|---|---|---|
| Head-mounted RGB video | First-person view from the actor’s own head position and gaze | Matches the viewpoint a robot’s head or chest camera will see at inference |
| Wrist-mounted video | Close-up view of hand-object contact during manipulation | Supervision for grasp, contact, and fine manipulation that a head camera alone can miss |
| Depth | Per-frame distance data aligned with the RGB streams | Geometry for contact and reach, not just appearance |
| 3D hand pose | Tracked hand and finger position over time | A measured action signal rather than one inferred after the fact |
| Action segments | Labeled start/end intervals with a verb-noun, e.g. "grasp mug" | Turns a continuous recording into supervised, temporally-localized training units |
05
Egocentric data collection vs third-person video vs teleoperation
| Method | Viewpoint captured | Typical limitation |
|---|---|---|
| Egocentric data collection | First-person, from a camera worn by the person doing the task | Requires dedicated capture hardware and a person performing the task on-site |
| Third-person / internet video | An external observer’s viewpoint, not the actor’s own | Viewpoint and hand-object geometry don’t match what a robot’s own camera will see |
| Teleoperation | The robot’s own onboard camera, driven remotely by a human operator | Captures the target embodiment directly but is slower to scale and still differs from a human’s own movement range |
06
Frequently asked questions
- What is egocentric data collection?
- It is the process of capturing first-person video and synchronized streams — depth, pose, and action labels — from a camera worn by the person performing a task, so the data shares the viewpoint a robot’s own camera would see.
- How is egocentric data collection different from egocentric video?
- Egocentric video is the resulting first-person footage; egocentric data collection is the process — the hardware, calibration, synchronization, consent, and labeling work — that produces it.
- How is egocentric data collection different from internet video?
- Internet or third-person video is recorded from an external observer’s viewpoint, which does not match what a robot’s own head or chest camera sees. Egocentric collection is captured from the actor’s own point of view specifically to close that gap.
- How is egocentric data collection different from teleoperation?
- Teleoperation records from the robot’s own onboard camera while a human operator drives it remotely, capturing the target embodiment directly. Egocentric collection instead records a human performing the task from their own point of view, which scales faster but still requires alignment work to be useful for robot training.
- What makes egocentric data collection usable for AI training?
- Calibrated capture hardware, sub-frame multi-stream synchronization, consent tracking, and labels recorded at time of capture — without these, footage is watchable but not reliably trainable.
- Who uses egocentric data collection?
- Teams building robot foundation models, vision-language-action models, and embodied-AI systems that need first-person training data matching their robot’s own camera viewpoint.
- Should I use a public egocentric dataset or a custom collection?
- Public datasets are a fast way to start, but a custom, on-demand collection lets you scope capture around the exact tasks, environments, and coverage a public corpus doesn’t already supply, built to a written spec.
- How do you start an egocentric data collection?
- Start with a written skill spec defining the task, environments, and conditions to capture, then move through a pilot batch before scaling — the same spec-first process Firsthand runs for custom egocentric data collection.
Related terms