Glossary
What Is Learning Reward Functions for Robotic Manipulation by Observing Humans?
Last reviewed: November 2026
Short answer
HOLD (Human Offline Learned Distances) is the method in “Learning Reward Functions for Robotic Manipulation by Observing Humans” (arXiv 2211.09019). From unlabeled human videos it learns a distance to a goal image, using temporal regression or time-contrastive embeddings, that serves as a dense robot reward. Firsthand is not affiliated.
01
What is HOLD, and what problem does it address?
HOLD is named in arXiv 2211.09019 by Minttu Alakuijala, Gabriel Dulac-Arnold, Julien Mairal, Jean Ponce and Cordelia Schmid (Inria, Google, Univ. Grenoble Alpes, Ecole Normale Supérieure); the project page lists ICRA 2023. The authors note that human and robot arms differ in morphology and that human actions are not observed unless specially recorded. HOLD therefore learns a task-agnostic reward from video alone, with no robot data and no task-specific human demonstrations.
02
Where does Firsthand fit next to HOLD?
HOLD is third-party research. Firsthand is not affiliated with the authors, and no named model, HOLD included, was trained on Firsthand data. Firsthand does not claim to use Something-Something v2, the Sawyer arm or Soft Actor-Critic. The site states custom egocentric collection, fast and on demand: first-person human video with calibrated stereo depth, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, with one task instruction per episode. Reward functions, goal images, robot data and trained policies are not Firsthand deliverables.
03
How does HOLD learn time-contrastive reward functions?
Per the paper, the goal is a distance between the current image and a goal image that tracks how many time steps an expert would need to reach the goal. The authors learn it by direct regression of that time gap (HOLD-R) or with a single-view time-contrastive embedding (HOLD-C), where frames close in time are pulled together and distant frames are pushed apart. The reward is the negated distance to the goal image, added to a sparse success reward.
04
What human videos did the HOLD authors use?
The authors train on Something-Something v2, which the paper describes as 220,847 crowd-sourced clips of 174 action classes, about 4 seconds long and mostly filmed on handheld devices. They say they do not use the class labels for the distance objectives, with one footnoted exception for pretraining a HOLD-R variant. The dataset licence is Not stated in the sources read.
05
How is the learned reward used to train a robot policy?
The authors train one policy per robot task with Soft Actor-Critic on image input, evaluated over 20 episodes, on simulated tasks including Pushing, Drawer Opening and Close Drawer. No robot data from the target environment is used to learn the reward.
06
What did the authors report, and what are the limits?
The authors report faster training than sparse reward alone (second table); Firsthand has not verified it. They state the distances are most accurate for short-horizon tasks and that Drawer needs an intermediate goal image.
07
How does HOLD compare with related entries on this site?
The third table sets HOLD beside four site entries. HOLD defines the task with a goal image and needs no robot demonstrations or task labels.
08
What would a HOLD-style approach need from custom collection?
Per the paper, it needs human manipulation videos, a goal image per robot task, a simulator or robot and a reinforcement learning stack, all the buyer’s own. The paper does not describe egocentric capture. Firsthand covers the human video side only, as custom collection on demand, confirmed on a Spec Pilot.
09
HOLD at a glance, as stated in the paper and project page
| Item | What the sources state |
|---|---|
| Paper | Learning Reward Functions for Robotic Manipulation by Observing Humans (arXiv 2211.09019) |
| Method name | Human Offline Learned Distances (HOLD) |
| Venue | ICRA 2023, per the project page |
| Input | Unlabeled human videos (Something-Something v2); no robot data |
| Two variants | HOLD-R (temporal regression) and HOLD-C (time-contrastive embeddings) |
| Task specification | A goal image |
| Policy learning | Soft Actor-Critic, simulated robot arm |
| Dataset licence | Not stated in the sources read |
| Firsthand relationship | None |
11
Results reported by the HOLD authors (simulation; not verified by Firsthand)
| Item | What the authors report |
|---|---|
| Sample efficiency | Up to 18x faster training on 5 simulated tasks when providing shaped rewards in sparse-reward tasks |
| Pushing | Sparse reward alone needed 10x more samples to reach the return of HOLD-C |
| Drawer Opening | Sparse reward alone needed more than 18x more samples to reach the return of HOLD-C |
| Close Drawer | HOLD solves the task without the sparse reward |
12
HOLD beside related entries on the site
| Site entry | What the entry states | How HOLD differs (sources read) |
|---|---|---|
| DVD | Classifies whether two videos show the same task, using task labels and a small robot dataset | Uses a goal image, with no task labels or robot demonstrations; the paper evaluates on DVD tasks |
| VIP | Pre-trains a representation on Ego4D human video; distance to a goal image gives a reward | Trains its own distance on Something-Something v2; the paper cites VIP as concurrent work |
| R3M | Pre-trains a frozen representation on Ego4D with time-contrastive learning | The paper uses R3M distances as a baseline and says HOLD-C needs no language descriptions |
| WHIRL | Turns one human video into a robot policy, improved by real-world trials | Learns a reusable reward from many videos |
13
Frequently asked questions
- What is HOLD?
- Per arXiv 2211.09019, HOLD (Human Offline Learned Distances) learns a task-agnostic, goal-image-conditioned reward for robotic manipulation from unlabeled human videos. It is third-party research.
- What are time-contrastive reward functions?
- HOLD-C embeds single frames so that frames close in time are near each other. Distance to a goal image in that space becomes the dense reward.
- What human data did HOLD train on?
- Something-Something v2, per the paper. The dataset licence is Not stated in the sources read.
- What did the authors report for HOLD?
- They report up to 18x faster training on 5 simulated tasks than sparse reward alone. Firsthand has not verified these figures.
- Was HOLD trained on Firsthand data?
- No. Firsthand is not affiliated with the authors, and no named model was trained on Firsthand data. Reward functions are not Firsthand deliverables.
- How can I request a fast, on-demand custom egocentric dataset for this research?
- Request a custom dataset with your tasks and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep.
Related terms
- DVD (Learning Generalizable Robotic Reward Functions from In-the-Wild Human Videos)
- VIP (Value-Implicit Pre-training)
- R3M
- WHIRL (Human-to-Robot Imitation in the Wild)
- HuDOR (Bridging the Human to Robot Dexterity Gap through Object-Oriented Rewards)
- Embodiment gap
- XIRL (Cross-embodiment Inverse Reinforcement Learning)
- GraphIRL (Graph Inverse Reinforcement Learning from Diverse Videos)
- Time-Contrastive Networks (TCN)
Sources
- Learning Reward Functions for Robotic Manipulation by Observing Humans (arXiv 2211.09019)
- HOLD project page (official, linked from the paper)
- The "something something" video database for learning and evaluating visual common sense (arXiv 1706.04261)
- Time-Contrastive Networks: Self-Supervised Learning from Video (arXiv 1704.06888)
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (arXiv 1801.01290)
- Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videos (DVD, arXiv 2103.16817)
- R3M: A Universal Visual Representation for Robot Manipulation (arXiv 2203.12601)
- VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training (arXiv 2210.00030)