Glossary
What Is the EgoBody Dataset?
Last reviewed: October 2026
Short answer
EgoBody is a published dataset of 3D human body shape and motion during two-person social interactions, introduced in arXiv 2112.07642. A HoloLens2 headset records egocentric streams while a multi-Kinect rig supplies SMPL-X ground truth, across 125 sequences. It is third-party research, not Firsthand’s.
01
What is the EgoBody dataset?
EgoBody is introduced in arXiv 2112.07642 by Siwei Zhang and co-authors from ETH Zürich and Microsoft, published at ECCV 2022, with an official project page and GitHub repository. The abstract says reasoning about social interactions from egocentric views depends on the body pose and motion of the interaction partner, and that research was hindered by a lack of datasets. Read: the abstract, the project page and the README; not the full paper text.
02
Where does Firsthand fit next to EgoBody?
EgoBody is third-party research. Firsthand is not affiliated with its authors and claims no use, training or compatibility with it. Firsthand captures real first-person human video, fast, on demand and to a written spec: 21 keypoints per hand with a visibility flag, per-frame 3D body joints, wrist trajectories, contact events and action segments, plus one task instruction per episode.
03
How was EgoBody captured, per the sources?
The abstract says Microsoft HoloLens2 headsets record egocentric RGB, depth, eye gaze, head and hand tracking. The headset is calibrated with a multi-Kinect rig, and SMPL-X body meshes are fitted to multi-view RGB-D frames. The project page says two subjects perform predefined interactions in indoor scenes, with 3 to 5 Azure Kinects and a synchronized HoloLens2 worn by one of them. Frame rates and resolutions are Not stated in the sources read.
04
What body shape, motion and interaction annotations does EgoBody provide, per the sources?
The README lists SMPL-X and SMPL body parameters for each frame, for both the interactee and the camera wearer, in the master Kinect’s coordinate system. It also lists OpenPose BODY_25 keypoints for the interactee in egocentric frames, 3D scene meshes and calibration files. Motion text labels come from the Motion-X dataset under a separate licence signing. Annotation accuracy figures are Not stated in the sources read.
05
What benchmarks or tasks does EgoBody define, per the sources?
The abstract proposes the first benchmark for 3D full-body pose and shape estimation of the social partner from egocentric views, and evaluates state-of-the-art methods. The README reports MPJPE, PA-MPJPE, V2V and PA-V2V for several baselines and shows that fine-tuning SPIN, METRO and EFT on the training set gives the lowest errors; the numbers are not reproduced here. The project page describes a workshop challenge on pose and shape from a single egocentric RGB image.
06
How does a social-interaction body-pose benchmark differ from custom real-world egocentric task capture?
EgoBody is a fixed, released dataset for estimating a partner’s body in indoor scenes, downloaded after signing a dataset licence whose terms are Not stated in the sources read. Firsthand is custom-collected: tasks, environments and signals are written into a spec and confirmed on a Spec Pilot. Firsthand claims no SMPL-X body meshes, no multi-Kinect third-person views and no annotation of an interaction partner, and it is not affiliated with EgoBody’s authors.
07
What egocentric data can you get custom-collected on demand?
Firsthand captures real human first-person video, with no synthetic or simulated data. Each episode carries ten streams aligned under 2 ms, including a head camera, two wrist cameras, depth, an IMU, and 3D hand and body pose. The median from signed spec to first validated batch is 48 hours where coverage is deep and three to four weeks where thin. Formats: RLDS, LeRobot v2.1, WebDataset, HDF5.
08
EgoBody capture setup, modalities and annotations, as stated in the official sources
| Item | What the official sources state |
|---|---|
| Publication | arXiv 2112.07642; ECCV 2022 (project page) |
| Egocentric device | Microsoft HoloLens2, worn by one subject |
| Egocentric streams | RGB, depth, eye gaze, head and hand tracking (abstract) |
| Third-person rig | 3 to 5 Azure Kinects with RGB-D, synchronized (project page) |
| Scale | 125 sequences, 36 subjects, 15 indoor scenes; 219,731 multi-view RGB-D frames and 199,111 egocentric RGB frames (project page) |
| Body annotations | SMPL-X and SMPL for interactee and camera wearer (README) |
| Motion text | Labels from the Motion-X dataset (project page) |
| Splits | Train, validation and test; 175,611 EgoSet_interactee frames where the interactee is visible (README) |
| Licence | Dataset licence must be signed to download; terms Not stated in the sources read |
| Firsthand relationship | None |
10
Benchmark metrics and methods named in the EgoBody README
| Item | What is stated |
|---|---|
| Task | 3D full-body pose and shape estimation of the social partner from egocentric views (abstract) |
| Metrics | MPJPE, PA-MPJPE, V2V, PA-V2V; units Not stated in the sources read |
| Baselines | CMR, SPIN, LGD, METRO, PARE, EFT |
| Fine-tuned | SPIN-ft, METRO-ft, EFT-ft, fine-tuned on the EgoBody training set |
| Finding | Fine-tuned rows show the lowest errors in the README table |
| Challenge | First phase: 3D pose and shape from an egocentric monocular RGB image (project page) |
11
What EgoBody’s sources describe beside what the Firsthand site states
| Aspect | EgoBody (sources read) | What the Firsthand site states |
|---|---|---|
| Origin | Fixed released dataset; two subjects in predefined interactions | Custom-collected to a written spec, on demand |
| Cameras | HoloLens2 egocentric RGB plus Azure Kinect third-person views | Head-mounted egocentric video plus two wrist cameras; no third-person views claimed |
| Body | SMPL-X and SMPL shape and pose, wearer and interactee | Per-frame 3D body joint coordinates; body shape meshes not claimed |
| Hands | Hand tracking from the headset (abstract) | 21 keypoints per hand with a per-joint visibility flag |
| Gaze | Eye gaze stream | A gaze_target field where the spec calls for it; sampling rate Not stated |
| Depth | HoloLens2 depth and Kinect RGB-D | Depth stream at 848 × 480 and 30 fps |
| Language | Motion text labels from Motion-X | One task instruction per episode |
| File layout | Image, CSV, NPZ and PKL files per the README | RLDS, LeRobot v2.1, WebDataset, HDF5 |
| Compatibility | Not claimed | Not claimed |
12
Frequently asked questions
- What is the EgoBody dataset?
- A dataset for human pose, shape and motion estimation from egocentric views during interactions in 3D scenes, introduced in arXiv 2112.07642.
- What was EgoBody captured with?
- A Microsoft HoloLens2 headset for egocentric streams, calibrated with a multi-Kinect rig that supplies the 3D ground truth, per the abstract.
- How big is EgoBody?
- The project page and README state 125 sequences, 36 subjects and 15 indoor scenes. Hour counts are Not stated in the sources read.
- Is EgoBody free to download?
- Download requires registering and signing the dataset licence. The licence terms are Not stated in the sources read.
- Does Firsthand provide body shape meshes or SMPL-X annotations?
- No. The site states per-frame 3D body joint coordinates beside hand pose, depth and RGB, and does not claim body shape meshes or third-person views.
- Does Firsthand use EgoBody, and can I request a custom dataset?
- Firsthand is not affiliated with the EgoBody authors and claims no use, training or compatibility. You can request a custom dataset, confirmed first on a Spec Pilot.
Related terms