Glossary
What Is MAPLE (Dexterous Manipulation Priors from Egocentric Videos)?
Last reviewed: November 2026
Short answer
MAPLE is a published method that trains a visual encoder on egocentric video to predict where the thumb and index finger contact an object and the 3D hand pose at contact. Its frozen features then feed policies for dexterous robot hands. It is from arXiv 2504.06084; Firsthand is not affiliated with the authors.
01
What is MAPLE, and what problem does it address?
MAPLE is introduced in “MAPLE: Encoding Dexterous Robotic Manipulation Priors Learned From Egocentric Videos” (arXiv 2504.06084) by Alexey Gavryushin, Xi Wang, Robert Jomar Malate, Chenyu Yang, Davide Liconti, René Zurbrügg, Robert Kevin Katzschmann and Marc Pollefeys. The arXiv record lists ETH Zurich; the project page lists ETH Zürich, Mimic Robotics and Microsoft Research. The paper expands the name as Encoding Manipulation Priors Learned from Egocentric Videos. The authors argue that general-purpose visual representations lack a focus on manipulation cues, and that earlier methods trained on egocentric video lack a focus on dexterous manipulation. Their hypothesis is that object contact points and grasping hand poses are a strong prior for dexterous tasks, so they train an encoder to predict exactly those from a single image.
02
Where does Firsthand fit next to MAPLE?
MAPLE is third-party research. Firsthand is not affiliated with the authors, and no named model, MAPLE included, was trained on Firsthand data. The site states custom egocentric collection to a written spec, fast and on demand, delivering first-person human video with 3D hand pose at 21 joints per hand and a per-joint visibility flag, calibrated stereo depth and 6-DoF IMUs at 200 Hz. The only language data tied to Firsthand capture is one task instruction per episode. Firsthand does not build or train models, and it does not deliver the paper’s contact-point labels, object masks, tokenised hand poses, encoder features, robot actions or policies.
03
How does MAPLE get training labels from egocentric video?
Per the paper, MAPLE draws its supervision automatically from existing egocentric video. The authors define a contact frame, the moment a hand first touches an object, and an earlier prediction frame in which the hand is still far from that object. From the contact frame they extract the 3D hand pose and the thumb and index fingertip contact points, then track those points back in time to the prediction frame.
The pipeline runs on Ego4D, which the paper describes as more than 3,000 hours of unscripted human activity. Following earlier protocols, the authors use a subset of approximately 1,590 hours and report approximately 82,100 training samples. They use off-the-shelf tools for the steps: the VISOR-HOS hand-object segmenter to find contact, HaMeR to estimate MANO hand poses, and the SAM-PT point tracker. The authors state that the method relies on no human-annotated data.
04
How does the MAPLE encoder learn, and how is it used?
The encoder is a ViT-B/16 initialised from DINO weights, producing 768-dimensional features. A small Transformer decoder, two layers with four attention heads each, reads those features from the prediction frame and predicts the two contact points and a tokenised 21-joint hand pose. Because the decoder sees only the encoder’s lower-dimensional embeddings, the authors say this bottleneck pushes the encoder to keep manipulation-relevant information.
The authors train two variants. MAPLE-LN updates only the LayerNorm parameters, and MAPLE-AP updates all parameters and adds a temporal contrastive loss. After training, the encoder is frozen. In simulation its features and proprioception feed an MLP policy; in the real-world experiments they feed a Diffusion Policy with a Diffusion Transformer.
05
What did the authors report, and what are the limits?
The authors evaluate on four tasks from the DAPG suite (open door, use hammer, rotate pen, relocate ball) and on three new MuJoCo tasks with an Adroit hand: pan, brush and clothes iron. Across these seven tasks they report that MAPLE-LN has the best mean success rate, 21.7, against 21.0 for DINO and 20.6 for MAPLE-AP, and they note that the mean for R3M is close. Real-world tests use three tasks, plush, pan and brush, with 16 trials per model and task. The authors report that MAPLE-LN has the highest overall performance and that MAPLE-AP slightly outperforms it on brush. These are the authors’ figures on their own setups; Firsthand has not verified them.
The authors list limits. They test one embodiment in simulation and one in the real world, so the effect of the robot is hard to estimate, and the extracted training data is noisy, with failure modes they call self-contact, premature contact and premature tracking termination. They name language conditioning and more diverse, possibly exocentric, human data as future directions.
06
What hardware, code and licence terms do the sources state?
For the real-world tests the paper names a Mimic robotic hand mounted on a Franka Emika Panda arm, with two external cameras, and it names Rokoko Smartgloves with the Coil Pro motion capture system for recording the robot demonstrations. Firsthand does not claim to use any of this hardware. The MAPLE training video is existing Ego4D footage, so the paper names no capture rig for it.
The paper says the authors will release their codebase, data extraction pipeline, extracted data and new benchmark. The official GitHub repository, as read for this entry, offers an encoder checkpoint download and a feature-extraction example, and says more code is coming. The licence for the code, checkpoints and extracted data is Not stated in the sources read, and so is the Ego4D licence, so check the official pages before any commercial use. No official Hugging Face page was found in the sources read.
07
How does MAPLE differ from HRP, R3M, HaMeR and DexWild?
MAPLE sits beside several entries on this site. HRP is the closest relative: it also mines affordances from Ego4D and fine-tunes an existing encoder, while R3M pre-trains on Ego4D with contrastive and language objectives. HaMeR is a tool inside the MAPLE labelling pipeline, DexWild co-trains robot policies on human hand data, and egocentric pretraining is the broader practice. The first table below sets them side by side.
08
What does Firsthand state it can supply for this kind of work?
MAPLE learns from Ego4D video that the authors processed with off-the-shelf tools. Firsthand’s stated scope is different: custom collection of first-person human video, depth, 3D hand pose and one task instruction per episode, confirmed first on a Spec Pilot. Contact-point labels, object masks, retargeted actions and robot trajectories are not Firsthand deliverables, and the site makes no claim that its data suits any MAPLE pipeline.
Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks. Billing is per validated hour. Formats are RLDS, LeRobot format or WebDataset under a buyer-owned commercial license.
09
MAPLE at a glance, as stated in the paper
| Item | What the paper states |
|---|---|
| Paper | MAPLE: Encoding Dexterous Robotic Manipulation Priors Learned From Egocentric Videos (arXiv 2504.06084) |
| Method name | MAPLE, Encoding Manipulation Priors Learned from Egocentric Videos; variants MAPLE-LN and MAPLE-AP |
| Training video | Ego4D (more than 3,000 hours); a subset of approximately 1,590 hours; approximately 82,100 training samples |
| Labels | Thumb and index contact points and 3D hand pose at contact, extracted automatically; no human-annotated data |
| Encoder | ViT-B/16 initialised from DINO weights, 768-dimensional features, frozen after training |
| Downstream policy | MLP in simulation; Diffusion Policy with a Diffusion Transformer in real-world tests |
| Robot hardware named | Mimic robotic hand on a Franka Emika Panda arm; demonstrations recorded with Rokoko Smartgloves and Coil Pro |
| Stated limits | One embodiment in simulation and one in the real world; noisy extracted training data |
| Code and data | Official project page and GitHub repository; licence Not stated in the sources read |
| Firsthand relationship | None; no named model was trained on Firsthand data |
11
MAPLE beside related entries on this site
| Site entry | What the entry states | How MAPLE differs (sources read) |
|---|---|---|
| Dexterous manipulation | Fine-grained, multi-finger manipulation such as regrasping, in-hand reorientation, precision pinches and tool use | MAPLE is a method for learning a visual prior for dexterous robot hands, tested in simulation and on a robotic hand |
| HaMeR hand mesh recovery | A transformer that reconstructs a 3D hand mesh from a single image by predicting MANO parameters | MAPLE uses HaMeR as an off-the-shelf tool to obtain hand poses for its labels |
| Egocentric pretraining | Pretraining a robot foundation model on large-scale first-person human video with action labels, before mid-training and post-training | MAPLE trains a vision encoder with automatically derived labels; the paper does not describe a foundation model |
| HRP | Mines hand, object and contact affordance labels from Ego4D and fine-tunes an existing visual representation to predict them | MAPLE also uses Ego4D and DINO weights; the paper says HRP samples contact points from bounding-box intersections, while MAPLE projects fingertips onto object masks |
| R3M | A visual representation pre-trained on Ego4D with time-contrastive learning and video-language alignment, used as a frozen module | The paper compares against R3M and says R3M relies on human-annotated language captions while MAPLE learns in an entirely self-supervised manner |
| DexWild | Collects human hand data in the wild and co-trains robot policies on it plus a smaller robot dataset | MAPLE collects no video; it reuses Ego4D and trains a frozen encoder |
12
What the MAPLE pipeline uses beside what the Firsthand site states
| Aspect | MAPLE (sources read) | What the Firsthand site states |
|---|---|---|
| Source footage | Existing Ego4D video, a subset of approximately 1,590 hours | Custom first-person human video, collected to a written spec |
| Language | Not used by the method as described; language conditioning is listed as future work | One task instruction per episode |
| Hand signal | MANO 3D hand poses estimated from video by an off-the-shelf model at contact frames | 3D hand pose at 21 joints per hand with a per-joint visibility flag |
| Contact labels | Thumb and index contact points from object masks and point tracking | Not delivered as MAPLE-style pixel contact-point labels |
| Depth and inertial | Not listed as inputs | Calibrated stereo depth and 6-DoF IMUs at 200 Hz |
| Robot actions | Policies trained on robot demonstrations recorded with gloves and motion capture | Not delivered; Firsthand captures first-person human video, not robot or teleoperation data |
| Training use | The authors’ own training and evaluation | No named model was trained on Firsthand data; no affiliation |
| Delivery | Public project page and repository; licence Not stated in the sources read | Spec Pilot, then billed per validated hour; custom collection, fast and on demand |
13
Frequently asked questions
- What is MAPLE?
- MAPLE is a published method (arXiv 2504.06084) that trains a visual encoder on egocentric video to predict hand-object contact points and 3D hand pose at contact. The features are then used by dexterous robot-hand policies. It is third-party research.
- What are the dexterous manipulation priors in MAPLE?
- Per the paper, the priors are where the thumb and index finger will contact an object and how the hand is posed at that moment. The encoder predicts both from a single earlier frame, so its features carry that information.
- Which dataset does MAPLE learn from?
- The authors run their extraction pipeline on Ego4D, which the paper describes as more than 3,000 hours of video. They use a subset of approximately 1,590 hours and report approximately 82,100 training samples. They use no human annotations.
- Did the authors test MAPLE on a real robot?
- Yes. The paper describes real-world plush, pan and brush tasks on a Mimic robotic hand mounted on a Franka Emika Panda arm, with 16 trials per model and task. The authors report that MAPLE-LN has the highest overall performance; Firsthand has not verified this.
- Is MAPLE open source, and can it be used commercially?
- The authors link an official project page and GitHub repository, which offers an encoder checkpoint and a feature-extraction example. The licence is Not stated in the sources read, so read the licence text on the official pages first.
- How is MAPLE different from HRP and R3M?
- All three learn visual representations from Ego4D. The paper says HRP samples contact points from bounding-box intersections, while MAPLE projects fingertips onto object masks, and that R3M relies on human-annotated language captions, while MAPLE is self-supervised.
- Was MAPLE trained on Firsthand data?
- No. Firsthand is not affiliated with the authors and claims no use, training or compatibility. Contact-point labels, object masks, encoder features, robot actions and policies are not Firsthand deliverables.
- How can I request a custom egocentric dataset for dexterous manipulation?
- Request a custom dataset with your objects, interactions and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep, confirmed first on a Spec Pilot.
Related terms
Sources
- MAPLE: Encoding Dexterous Robotic Manipulation Priors Learned From Egocentric Videos (arXiv 2504.06084)
- MAPLE project page (official)
- algvr/maple (official GitHub, linked from the project page)
- Ego4D: Around the World in 3,000 Hours of Egocentric Video (arXiv 2110.07058)
- Ego4D (official site)
- Reconstructing Hands in 3D with Transformers (HaMeR, arXiv 2312.05251)