What Is EgoZero (Robot Learning from Smart Glasses)?

EgoZero is named in EgoZero: Robot Learning from Smart Glasses (arXiv 2505.20290) by Vincent Liu, Ademi Adeniji, Haotian Zhan, Siddhant Haldar, Raunaq Bhirangi, Pieter Abbeel and Lerrel Pinto. The authors argue that robot policies still lag far behind basic human capabilities, and that the data bottleneck stems less from a shortage of human labor than from the unresolved challenge of capturing and representing human behavior for robot learning. Their question is whether robots can learn zero-shot manipulation skills from only egocentric, in-the-wild human data. They say EgoZero removes the need for teleoperation, calibration or additional wearables during data collection. For the broader ideas, see the site’s imitation learning and cross-embodiment learning entries.

Firsthand is not affiliated with the EgoZero authors and makes no claim that any model, EgoZero included, was trained on Firsthand data. EgoZero records with Project Aria smart glasses; Firsthand does not claim to use that hardware. Firsthand captures real human egocentric video, custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license, with one task instruction per episode. Point-based state or action representations, hand-pose retargeting outputs, robot trajectories, action labels, reward labels, pretrained representations, embeddings, teleoperation data, paired human-robot video, synthetic data and language annotations beyond that instruction are not Firsthand deliverables.

Per the paper, the authors record human demonstrations with Project Aria smart glasses and compress each into a morphology-agnostic state and action, both expressed as compact sets of 3D points in the wearer’s egocentric frame. The action is the thumb and index fingertip coordinates plus a gripper closure, with grasps detected by thresholding the distance between thumb and index. The state is a set of object points. The glasses provide neither stereo nor depth, so the authors triangulate expert-labeled 2D points over the camera trajectory, using the glasses’ SLAM extrinsics and the CoTracker3 point tracker. A closed-loop Transformer policy is then trained with behavior cloning on those state-action pairs.

At inference, the authors use an iPhone as a stationary egocentric view that supplies accurate depth, map the labeled points with Grounding DINO and DIFT, and execute on a Franka Panda gripper robot through inverse kinematics. Their ablation says random 3D augmentations are necessary for in-the-wild transfer.

The authors state they use the Project Aria fisheye RGB camera and two SLAM cameras, plus the glasses’ Machine Perception Services, which supply 6DoF hand poses, camera intrinsics and camera extrinsics. A demonstration is a sequence of RGB images, 6DoF palm poses and 6DoF camera extrinsics. To recover finger positions they add HaMeR, a hand model that gives a 21-keypoint hand, composing its local hand shape with the glasses’ hand pose. The authors note that although there are three visual cameras, they have little field-of-view overlap, which makes stereo triangulation unreliable.

The authors report zero-shot transfer to a Franka Panda gripper robot with a 70% success rate over 7 manipulation tasks, trained on 100 human demonstrations per task, which they describe as 20 minutes of data collection per task, and with no robot data. The tasks are open oven door, put bread on plate, sweep board with broom, erase board, sort fruit into a bowl, fold towel and insert book in shelf. In their baselines, a policy learned directly from images, and variants without 3D augmentations or without triangulated depth, recorded zero successes. These are the authors’ figures on their setup; Firsthand has not verified them.

The authors also list limits. The policy is bounded by the accuracy of its 3D point inputs, and the correspondence model is the largest source of inference error. Triangulation needs stationary objects and enough camera movement, so objects cannot be tracked. Hand models add slight inaccuracies, leaving action labels with 1–2 cm of error, which the authors say prevents high-precision tasks.

The first table sets EgoZero beside five site entries, summarising what those entries state. The main difference is representation: EgoZero trains on 3D points rather than images and uses no robot data, while EgoMimic co-trains with robot data and HAT works with humanoid data.

Per the paper, it needs egocentric human demonstrations with camera pose and hand pose, expert-labeled points on the objects, point-tracking and hand-model tools, and a robot with a calibrated stationary camera for inference. The authors collected demonstrations in 2–3 environments per task with several demonstrators.

Firsthand states real human egocentric capture, depth from a calibrated stereo rig, 3D hand pose at 21 joints per hand, and one task instruction per episode. That is a note about format, not a compatibility claim. EgoZero does not use stereo; it derives object depth by triangulating over the camera trajectory, so Firsthand’s stereo depth belongs to the site’s general capture offer, not to this method. Labeled points, tracking, retargeting and robot rollouts would come from tools the buyer runs. No off-the-shelf dataset for this is stated here, so this is custom collection: request on-demand collection and confirm it on a Spec Pilot, with a median of 48 hours to a first validated batch where coverage is deep and three to four weeks where it is thin.

What is EgoZero?

EgoZero is a published system (arXiv 2505.20290) that learns robot manipulation policies from egocentric human demonstrations recorded with Project Aria smart glasses, using zero robot data. It is third-party research.

What hardware does EgoZero use?

Per the paper, Project Aria smart glasses record the human demonstrations. At inference an iPhone supplies a stationary egocentric view and the policy runs on a Franka Panda gripper robot. Firsthand does not claim to use that hardware.

Does EgoZero need robot data?

No. The authors state it learns from human demonstrations alone, with zero robot data, using point-based states and actions.

What did the authors report for EgoZero?

They report 70% zero-shot success on a Franka Panda gripper robot over 7 manipulation tasks, trained on 100 human demonstrations per task. These are the authors’ figures; Firsthand has not verified them.

Was EgoZero trained on Firsthand data?

No. Firsthand is not affiliated with the authors and claims no use, training or compatibility. Point-based representations, robot trajectories, action labels and paired human-robot video are not Firsthand deliverables.

How can I request a custom egocentric dataset for human-to-robot learning?

Request a custom dataset with your objects, interactions and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep.