What Is the EgoDex Dataset?

EgoDex is introduced in EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video (arXiv 2505.11709). The authors frame the problem as data scarcity for imitation learning: unlike text and 2D vision, there is no Internet-scale corpus for dexterous manipulation. They propose egocentric human video with paired 3D hand pose as a passively scalable middle path between robot teleoperation and unstructured web video.

EgoDex is third-party research. Firsthand is not affiliated with its authors, and makes no claim that EgoDex uses Firsthand data or hardware or that any model was trained on Firsthand data. The site states custom egocentric collection to a written spec, fast and on demand, with 3D hand pose at 21 joints per hand, calibrated stereo depth and 6-DoF IMUs at 200 Hz.

Per the paper, all data comes from one consumer headset with high-resolution passthrough cameras and on-device tracking. Collectors worked with bare hands and no additional hardware, and the cameras record what the collector sees without a pose offset. Recording is organised in sessions of roughly 10 to 15 minutes, with episode boundaries marked by pausing and resuming the recording app.

The paper names three task types: reversible tasks that come in inverse pairs, reset-free tasks whose end state falls within their own start distribution, and reset tasks that need the scene restored after each demonstration. It says the first two give a higher yield because resets are not recorded.

Each episode carries 1920×1080 RGB video at 30 Hz, camera intrinsics and extrinsics, 3D position and orientation of upper-body joints including 25 joints per hand, per-joint confidence values between 0 and 1, and a natural language description. A confidence of zero means the joint is fully occluded.

The task metadata was written by data collectors, and the paper says it combined those fields into a single description using a language model. The full dataset takes 2.0 TB on disk after video compression.

The authors set two benchmarks, dexterous trajectory prediction and inverse dynamics, with 1% of the data held out as a fixed test set. They report that prediction accuracy falls as the horizon grows and that performance improves with dataset size. Those are the authors’ results on their own baselines, and Firsthand quotes no numbers from them.

The paper also lists robotics, perception, video generation and egocentric world modelling as possible uses, and makes no claim about any specific deployed robot.

The paper says scene diversity is limited to tabletop environments, because the focus is behavioural diversity. It also says the hand annotations are themselves model predictions, so they can be imperfect under heavy occlusion, such as towel folding, or at very high speed.

Those limits matter when a program needs a different environment or a verified per-joint label. The confidence values are the dataset’s own signal for that.

When a program needs tasks, environments or objects that a published dataset does not cover, specified in writing. Firsthand does not build or train models, and this page makes no accuracy comparison between EgoDex and custom capture.

Confirm the spec on a Spec Pilot, a small fast first batch. Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks. Billing is per validated hour.

What is the EgoDex dataset?

A public egocentric dataset of dexterous human manipulation, with 1080p 30 Hz video and 3D hand and finger tracking recorded at capture time, from the paper arXiv 2505.11709.

How big is EgoDex?

The paper states 829 hours, 90 million frames, 338,000 episodes and 194 tabletop tasks, taking 2.0 TB on disk.

What hand data does EgoDex include?

3D position and orientation of upper-body joints, including 25 joints per hand, at 30 Hz, with a confidence value from 0 to 1 per joint. The paper says the annotations are model predictions and can be imperfect under heavy occlusion.

Does EgoDex cover environments beyond tables?

The paper says scene diversity is limited to tabletop environments, and that its focus is diversity in manipulation behaviours.

Does Firsthand use EgoDex?

No. Firsthand is not part of it or affiliated with its authors, and makes no claim that EgoDex uses Firsthand data or hardware or that any model was trained on Firsthand data.

Can Firsthand collect dexterous manipulation data to a custom spec?

The site states custom egocentric collection to a written spec, fast and on demand, with 21-joint 3D hand pose, calibrated stereo depth and 200 Hz IMUs, confirmed on a Spec Pilot.