What Is HRP (Human Affordances for Robotic Pre-Training)?

HRP is named in HRP: Human Affordances for Robotic Pre-Training (arXiv 2407.18911), submitted on 26 July 2024 by Mohan Kumar Srirama, Sudeep Dasari, Shikhar Bahl and Abhinav Gupta of Carnegie Mellon University. The arXiv comments say it was accepted to Robotics: Science and Systems 2024. The authors argue that a robot needs a vision representation that helps it predict good actions, that learning one from robot data alone is prohibitively expensive, and that earlier representations trained on human video do not reliably beat a simple ImageNet baseline. Their answer is to add explicit hand and object supervision, mined from human video, to an existing encoder. For the broader idea, see the site’s affordance entry.

Firsthand is not affiliated with the HRP authors and makes no claim that any model, HRP included, was trained on Firsthand data. Firsthand captures real human egocentric video, custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license, with one task instruction per episode. Affordance labels, contact-point labels, hand-pose or object-box labels, post-contact trajectories, heatmaps, pretrained representations, embeddings, reward labels, robot trajectories, teleoperation data, paired human-robot video, synthetic data and language annotations beyond that instruction are not Firsthand deliverables.

Per the paper, HRP is a semi-supervised pipeline in two stages. First it extracts hand-object affordance information from human video with off-the-shelf tracking and detection models, meaning which objects can be grasped and how a hand approaches them. Then it distils those labels into an existing representation network before the policy training stage. The authors say this injects useful information into the vision encoder while keeping the flexibility of self-supervised pre-training, because all labels are generated automatically and the encoder can be slotted into downstream policies by fine-tuning.

The paper describes three label types. Contact locations are points where the hand will touch an object in the future, found from the first frame of contact using a hand-object interaction detector and skin segmentation, then projected back with a homography and summarised by a Gaussian mixture model with five modes. Future hand poses are the 2D wrist position a fixed number of frames ahead, from a hand detector. Active object labels are the bounding box of the object being interacted with.

The authors take about 150K from Ego4D human-object interaction videos originally sampled for R3M, and the appendix describes roughly 1,200 source videos cut into clips of 1 to 30 seconds. Because per-frame detections are noisy, they apply a filter. These labels are inferred by software from ordinary video, not recorded by a sensor, so detector errors from occluded hands or cluttered scenes flow into the supervision.

HRP starts from an existing pre-trained encoder, a ViT-B with patch size 16 in the paper, adds three small prediction heads and trains them with L2 regression losses on the contact, hand and object labels. Because the affordance dataset is far smaller than the data the original encoder saw, the authors fine-tune only the normalisation layers and keep the other weights frozen. The project page says LayerNorm-only fine-tuning is almost always better than fine-tuning all weights.

The resulting representation is then used for behaviour cloning. Per the paper, each real task used 50 expert demonstrations collected by teleoperation, and the policy is a small MLP on top of the encoder that predicts end-effector and gripper velocity.

The authors apply HRP to six baseline representations: ImageNet MAE, Ego4D MAE, CLIP, DINO, MVP and VC-1. The abstract reports more than 3000 robot trials and a boost of at least 15% on 5 real-world tasks across three robot morphologies, including a dexterous hand, and three camera views. A figure caption reports their best representation beating six baselines by at least 20%, and the contributions list reports a 20% gain for diffusion policy. These are the authors’ figures on their setups; Firsthand has not verified them.

Caveats come from the sources themselves. The project page says the object and hand losses were critical while the contact loss mattered significantly only for the Ego4D base model. The appendix simulation table shows HRP improving some base representations and lowering others on a five-task MetaWorld average, so the gain is not uniform across every setting.

The first table sets HRP beside five site entries, summarising what those entries state. The short version: R3M, VC-1 and MVP train encoders with self-supervised or contrastive objectives, VRB predicts contact points and a trajectory as its deliverable, and HRP uses hand, object and contact labels as fine-tuning targets for an existing encoder. HRP also lists VC-1 and MVP among its baselines.

Per the paper, it needs human video with hand-object interaction, detectors to turn that video into labels, and robot demonstrations for the final behaviour-cloning stage. Firsthand states real human egocentric capture, depth from a calibrated stereo rig, 3D hand pose and contact events (see the affordance entry), with one task instruction per episode. That is a note about format, not a compatibility claim: any affordance labels would come from tools the buyer runs.

Ego4D is accessed under its own licence agreement; see the site’s Ego4D entry and confirm the current terms. For real human demonstrations that public video does not cover, request on-demand custom collection and confirm it on a Spec Pilot, with a median of 48 hours to a first validated batch where coverage is deep and three to four weeks where it is thin.

What is HRP (Human Affordances for Robotic Pre-Training)?

A published method (arXiv 2407.18911, RSS 2024) that fine-tunes an existing visual encoder to predict hand, object and contact affordance labels mined from human video, then uses it for robot policy learning. It is third-party research.

What are human affordance labels from egocentric video in HRP?

Per the paper, future contact points, future hand poses and active object bounding boxes, extracted from Ego4D video by off-the-shelf detectors rather than annotated by hand.

How does HRP differ from VRB?

VRB predicts contact points and a post-contact trajectory as its output. HRP uses contact, hand and object labels as targets to fine-tune an existing encoder, which is then used for behaviour cloning.

Was HRP trained on Firsthand data?

No. Firsthand is not affiliated with the authors and claims no use, training or compatibility. Affordance labels, hand-pose or object-box labels and pretrained representations are not Firsthand deliverables.

How can I request a custom egocentric dataset for representation fine-tuning?

Request a custom dataset with your tasks, environments and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep.