What Is HaMeR (Hand Mesh Recovery)?

HaMeR is introduced in Reconstructing Hands in 3D with Transformers (arXiv 2312.05251) by Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey and Jitendra Malik. The abstract presents it as an approach that reconstructs hands in 3D from monocular input, with a fully transformer-based architecture and, in the authors’ words, significantly increased accuracy and robustness compared to previous work. The abstract says the key to its success lies in scaling up both the training data and the capacity of the network. The README cites the paper as a CVPR 2024 publication, and the repository lists an MIT License. The official sources read are the paper, the project page and the GitHub repository. The Hugging Face Space could not be loaded when this entry was written, so it is not linked. The sources read describe HaMeR as a reconstruction model, not as a dataset or robot data.

HaMeR is third-party research. Firsthand is not affiliated with its authors or their institutions, makes no claim that any model was trained on Firsthand data, and makes no claim that HaMeR is part of its own pipeline. Firsthand captures first-person human video, collected fast, on demand and to a written spec. It is not robot or teleoperation data. The fields the site states are head and wrist camera video, depth derived from a calibrated stereo rig, 3D hand pose at 21 keypoints per hand with a per-joint visibility flag, action segments, and 6-DoF IMUs at 200 Hz on one clock. The only language field is one task instruction per episode.

The project page describes HaMeR as an approach for Hand Mesh Recovery from a single image. The inputs are an image of a hand, a bounding box around it and the hand side, left or right. A deep network reconstructs the hand in 3D as a MANO mesh: it predicts the MANO model parameters, which are used to get the 3D hand mesh. The abstract calls the network a large scale Vision Transformer, and the project page calls the design fully transformer-based.

The project page says the approach is applied on each frame independently, yet recovers temporally smooth results, and that its side-view videos are estimated per frame without additional smoothing. The README says users download trained models with a script and must register on the MANO website to download the MANO model, of which only the right hand model is required. The parameter count and loss function are Not stated in the sources read. The README install steps include a ViTPose package under third-party, and how the backbone uses it is Not stated in the sources read.

The abstract says HaMeR combines multiple datasets that contain 2D or 3D hand annotations for training. Which datasets, how many images and how they are weighted are Not stated in the sources read. The README training command refers to a data configuration called mix_all and a script that downloads the training data, and the contents of that mix are Not stated in the sources read.

For evaluation the abstract says the final model consistently outperforms previous baselines on popular 3D hand pose benchmarks, and the README names FreiHAND, HO-3D and HInt. The numerical results are Not stated in the sources read, and this page reports none. To test non-controlled settings the authors annotated existing in-the-wild datasets with 2D hand keypoints, a new set called HInt. The project page says HInt samples frames from New Days of Hands, EpicKitchens-VISOR and Ego4D, and that because the frames come from video they capture more natural interactions of the hands. The README says the HInt annotations are released, and it reports a 2nd place award in the Ego-Pose Hands task of the Ego-Exo4D Challenge in June 2024.

Keypoint hand pose estimation returns a skeleton, usually joint positions per frame. HaMeR returns a 3D hand mesh built from predicted MANO parameters. HInt itself is annotated with 2D keypoints, so the two outputs sit side by side even in the authors’ own evaluation set, though how results are scored is Not stated in the sources read.

The comparison table sets HaMeR beside what the site states for its own hand pose. They answer different questions and this page ranks neither. The project page also compares HaMeR with FrankMocap and Mesh Graphormer, saying those baselines fail under occlusions, truncations or unexpected illumination and are significantly more jittery. Whether HaMeR outputs are in metric units is Not stated in the sources read, and whether Firsthand delivers a mesh or MANO parameters is Not stated.

The project page says HaMeR is particularly robust to occlusions, truncations, different skin colors, hands wearing gloves and hands interacting with other hands or objects. It also lists limitations, shown in the failure-mode table. Failure rates are Not stated in the sources read.

HInt draws on frames from video datasets that include Ego4D and EpicKitchens-VISOR, and the project page stresses hands interacting with objects. The sources read report no results for a particular head-mounted camera setup, and how a wearer’s viewpoint affects accuracy is Not stated in the sources read.

Running a mesh model on a recording gives a prediction per frame. A delivered dataset is a different thing: Firsthand states 3D hand pose as 21 keypoints per hand in metric units in the head-camera frame, aligned to depth and RGB, with a visibility flag per joint that a pipeline can filter on. Mesh or MANO output is Not stated as a Firsthand deliverable. If a program needs it, put it in the spec and confirm it on a pilot.

HaMeR is a model, not a data source, and its weights, evaluation sets and annotations come from its authors. When a program needs hand data from particular tasks, environments or people, write them into a spec. The site states that both hands stay in frame with per-hand 3D pose and visibility flags. Confirm the spec on a Spec Pilot, a small fast first batch. Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks. Billing is per validated hour, and delivery is in the format your training stack already reads: RLDS, LeRobot, WebDataset or HDF5.

What is HaMeR?

A transformer-based model for Hand Mesh Recovery. Given a single image of a hand, a bounding box and the hand side, it predicts MANO parameters that give a 3D hand mesh. It comes from the CVPR 2024 paper Reconstructing Hands in 3D with Transformers.

What does HaMeR output?

Per the project page, it predicts the MANO model parameters, which are used to get the 3D hand mesh. Whether the output is in metric units is Not stated in the sources read.

What data was HaMeR trained on?

The abstract says multiple datasets that contain 2D or 3D hand annotations. The individual datasets and their sizes are Not stated in the sources read. The authors also annotated frames from New Days of Hands, EpicKitchens-VISOR and Ego4D with 2D keypoints, a set called HInt.

What is HInt?

A dataset of 2D hand keypoint annotations on frames sampled from New Days of Hands, EpicKitchens-VISOR and Ego4D, introduced to evaluate HaMeR in non-controlled settings. The README says the annotations are released.

Can HaMeR be used commercially?

The repository lists an MIT License. The MANO model is downloaded separately after registering on its website, and the terms for it and for the trained models are Not stated in the sources read, so read each licence yourself. This is not legal advice.

Does Firsthand use HaMeR, and can I request a custom dataset?

Firsthand is not affiliated with the HaMeR authors, makes no claim that any model was trained on Firsthand data, and makes no claim that HaMeR is part of its pipeline. You can request a custom egocentric dataset: write the tasks, fields and format into a spec, confirm it on a Spec Pilot, and billing is per validated hour. The only language field is one task instruction per episode.