What Is EgoMimic?

EgoMimic is introduced in the paper EgoMimic: Scaling Imitation Learning via Egocentric Video (arXiv 2410.24221), with an official project page and GitHub repository. The authors open with a data problem: the scale and diversity of demonstration data required for imitation learning is a significant challenge. Their answer is a full-stack framework with four parts: a system to capture human embodiment data, a low-cost bimanual manipulator that minimizes the kinematic gap to that data, cross-domain data alignment techniques, and an imitation learning architecture that co-trains on human and robot data. They contrast this with prior work that only extracts high-level intent from human videos. EgoMimic instead learns one unified policy from both sources, so the human data supervises the same network that controls the robot.

EgoMimic is third-party research. Firsthand is not affiliated with its authors or their institutions, and makes no claim that EgoMimic uses Firsthand data or hardware or that any model was trained on Firsthand data. The site does not state that Firsthand captures with Project Aria glasses. It states custom egocentric collection to a written spec, fast and on demand, delivering first-person human video with 3D hand pose at 21 joints per hand and a per-joint visibility flag, calibrated stereo depth, 6-DoF IMUs at 200 Hz and a separate tactile glove tier. Firsthand captures first-person human video, not robot or teleoperation data.

Per the paper, the human data system is built on Project Aria glasses, head-worn devices for capturing multimodal egocentric data, described as an ergonomic glasses form factor that weighs only 75g. EgoMimic uses the front-facing wide field-of-view RGB camera for visual observation and two mono-color scene cameras for device pose and hand tracking. The side-facing cameras keep tracking hands when they leave the main camera’s view, which the authors tie to people moving their head and gaze ahead of their hands.

The paper uses the Aria Machine Perception Service to estimate 3D poses of both hands, and visual-inertial SLAM to obtain the glasses pose in the world frame. Per the appendix, human data is recorded at 30 Hz and converted into an hdf5 file compatible with robomimic training. The repository includes a script that processes Aria recordings into that format, with an optional SAM masking step described in its data processing guide.

Per the paper, EgoMimic bridges three gaps between the two sources. It transforms hand and robot end-effector trajectories into camera-centered stable reference frames, applies Gaussian normalization separately to hand and robot poses and actions, and masks out the hand and the robot with SAM, overlaying a red line to indicate end-effector direction. The authors also mount a second pair of Aria glasses as the robot’s main camera to narrow the camera-to-camera gap.

The policy builds on ACT. All parameters are shared except two shallow input and output heads. A pose loss supervises both human and robot data, while the joint action loss supervises only robot data. The appendix explains why: Aria only records hand pose, so human data lacks information for the grasping action. The paper describes its robot as two 6-DoF ViperX 300 S arms with Intel RealSense D405 wrist cameras, and says the need to predict both pose and joint actions is specific to that hardware.

In its baseline comparison the paper removes goal conditioning from Mimicplay because EgoMimic is designed for single-task policies. Any language input for EgoMimic is Not stated in the sources read.

The abstract says that adding 1 hour of additional hand data is significantly more valuable than 1 hour of additional robot data. The paper shows this on one task, Continuous Object-in-Bowl: EgoMimic trained on 2 hours of robot data plus 1 hour of hand data is compared with ACT trained on 3 hours of robot data. The paper states that one hour of human data yields 1400 demonstrations, compared with 135 demonstrations from an hour of robot data. It also notes that tasks like Groceries were slower to collect human data for because of resetting time.

The authors attach caveats. They state that EgoMimic at 2 hours of robot data outperforms ACT at 2 hours of robot data, so some of the improvement is attributed to architecture. Their third hypothesis is conditional: given sufficient initial robot data, collecting additional human data is more valuable than additional robot data. Across its three real-world tasks the paper reports a relative improvement in score of 34-228% over ACT, and names generalizing to new robot embodiments and to behaviors demonstrated only in human data as future work. This page draws no general rule about how much human data any other policy needs.

EgoMimic is a research framework: the authors’ own glasses, their own paired robot, and processing scripts released under the MIT License. Custom egocentric capture is a data service: paid, consented contributors recording to a written spec, with the fields and format set in advance. The two answer different questions, and this page makes no ranking between them.

The comparison table below sets only what the official sources state against what the site states. Where a source is silent, the cell says so. The clearest difference is the robot half. EgoMimic pairs human data with teleoperated robot data, while Firsthand delivers first-person human video only.

When a program needs tasks, environments or people that an existing recording setup does not cover, specified in writing. A team building its own human-plus-robot co-training pipeline would supply the robot half itself, because Firsthand does not deliver robot or teleoperation data and does not build or train models. If contact signals matter, ask for the separate pilot-ready tactile glove tier, time-synced to a head-mounted stereo camera by hardware timestamps and an IMU.

Confirm the spec on a Spec Pilot, a small fast first batch. Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks. Billing is per validated hour, and delivery is in the format your training stack already reads: RLDS, LeRobot, WebDataset or HDF5.

What is EgoMimic?

A full-stack framework from a 2024 paper that co-trains one manipulation policy on egocentric human video with 3D hand tracking and on teleoperated robot data.

What is human embodiment data in EgoMimic?

Per the paper, egocentric human videos paired with 3D hand tracking, captured with Project Aria glasses while a person performs a task with their own hands, without a robot.

Does EgoMimic replace robot data?

No. The framework co-trains on human and robot data. The paper says its hypothesis is that, given sufficient initial robot data, additional human data is more valuable than additional robot data.

Is EgoMimic open source?

The official GitHub repository provides data processing and training code for Aria human data and teleoperated robot data, and lists the MIT License. Dataset licence terms are Not stated in the sources read.

Does EgoMimic take language instructions?

Not stated in the sources read. The paper describes EgoMimic as designed for single-task policies.

Does Firsthand use EgoMimic or Project Aria glasses?

No. Firsthand is not part of EgoMimic or affiliated with its authors, and makes no claim that EgoMimic uses Firsthand data or hardware or that any model was trained on Firsthand data. The site does not state that Firsthand captures with Aria glasses.

Does Firsthand deliver robot or teleoperation data?

No. Firsthand captures first-person human video. Mapping it to your robot, and collecting the robot half of any co-training setup, is a step in your own pipeline.

Can I request a custom egocentric dataset?

Yes. Write the tasks, environments, fields and format into a spec, confirm it on a Spec Pilot, and billing is per validated hour.