Glossary
What Is DVD (Learning Generalizable Robotic Reward Functions from In-the-Wild Human Videos)?
Last reviewed: November 2026
Short answer
DVD, the Domain-agnostic Video Discriminator, is the method in “Learning Generalizable Robotic Reward Functions from ‘In-The-Wild’ Human Videos” (arXiv 2103.16817, RSS 2021). It trains a discriminator to judge whether two videos show the same task, using a few robot demos plus many human videos, and the score serves as a robot reward. Firsthand is not affiliated.
01
What is DVD, and what problem does it address?
DVD is named in Learning Generalizable Robotic Reward Functions from “In-The-Wild” Human Videos (arXiv 2103.16817) by Annie S. Chen, Suraj Nair and Chelsea Finn of Stanford University, published at Robotics: Science and Systems. The authors want a robot to acquire a measure of task success, a reward, that generalizes across environments, tasks and objects while depending only on on-board sensor observations such as RGB images. They note that collecting high-quality robot interaction data at scale remains an open challenge, whereas in-the-wild human videos, for example from YouTube, show people doing many tasks in diverse settings.
02
Where does Firsthand fit next to DVD?
Firsthand is not affiliated with the DVD authors. No named model, DVD included, was trained on Firsthand data, and Firsthand does not claim to use any hardware or software named in the sources: they name the WidowX200 robot, a Sawyer arm in a MuJoCo simulation built on Meta-World, and the Something-Something-V2 dataset, which Firsthand does not supply. Firsthand captures real human egocentric video, custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license, with one task instruction per episode. Reward functions, reward or value labels, robot trajectories, action labels, simulation environments, digital twins, co-training mixtures, pretrained representations, embeddings, synthetic data and language annotations beyond that instruction are not Firsthand deliverables.
03
How does DVD learn a reward from videos?
Per the paper, DVD trains a classifier that takes two videos and predicts whether they complete the same task. Training batches are triplets: two videos labelled with the same task and one labelled with a different task, drawn from a mix of human and robot videos. The authors say this forces the model to learn the functional similarity between videos from very different visual domains.
The reward is a similarity score between a human demonstration video, which specifies the task, and the robot’s own behaviour. A pretrained video encoder, fixed during training, feeds a fully connected network that outputs the score. The authors state that the reward is inferred only from visual observations and assumes no actions or low-dimensional states from the human or robot data.
04
What video does DVD train on?
The human side is the existing Something-Something-V2 dataset, which the paper describes as 220,837 videos in 174 classes. Depending on the experiment the authors use up to 15 human tasks, each with 853 to 3,170 videos, and place no constraint on viewpoint, background or quality. The robot side is small: 120 simulated demonstrations of 3 tasks, and 80 demonstrations of each of 2 tasks on the real robot, which the authors say were collected by scripted or model-predictive routines.
The dataset licence is Not stated in the sources read.
05
How is the learned reward used to choose actions?
The authors pair DVD with visual model predictive control. An action-conditioned video prediction model, trained with SV2P, imagines the future for many sampled action sequences. The cross-entropy method then picks the sequence whose predicted video DVD scores as most similar to the human demo.
06
What did the authors report, and what are the limits?
The authors report that training on diverse human videos gives a 15 to 20 percent absolute improvement in downstream task success over using robot data alone, including for tasks unrelated to the target. On the real WidowX200 they report 20 trials per task in an unseen environment, with DVD trained on 6 human tasks succeeding over 65 to 70 percent of the time against at most 40 percent for robot-only. These are the authors’ figures; Firsthand has not verified them.
The paper states limits: it learns a reward, not a policy or a dynamics model; it needs some robot demos and task labels for all videos; and it is tested only on coarse tasks that do not need fine-grained manipulation. The authors report that success falls 6 percent with 20 robot demos per task and 13 percent with none.
07
How does DVD compare with related entries on this site?
The comparison table sets DVD beside five site entries. The main difference is that DVD learns a task-conditioned reward from labelled video pairs, where the entries it is compared with pretrain a representation, imitate one video, or predict hand motion.
08
What would a DVD-style approach need from custom collection?
Per the sources read, it needs labelled human videos of the target tasks, a small set of robot demonstrations, a video encoder, a video prediction model and a robot or simulator, all the buyer’s own. Firsthand states one task instruction per episode and no other labels, so any per-video task-class scheme is the buyer’s own. The paper does not state a need for stereo depth, hand pose or IMUs.
No off-the-shelf dataset for this is stated here, so this is custom collection: request on-demand collection and confirm it on a Spec Pilot, with a median of 48 hours to a first validated batch where coverage is deep and three to four weeks where it is thin.
09
DVD at a glance, as stated in the paper, project page and code repository
| Item | What the sources state |
|---|---|
| Paper | Learning Generalizable Robotic Reward Functions from “In-The-Wild” Human Videos (arXiv 2103.16817) |
| Method name | Domain-agnostic Video Discriminator (DVD) |
| Authors | Annie S. Chen, Suraj Nair, Chelsea Finn |
| Institution | Stanford University |
| Venue | Robotics: Science and Systems, 2021, per the project page |
| Core idea | Train a discriminator to classify whether two videos perform the same task; use its score as a reward |
| Human input | Something-Something-V2 videos with task labels |
| Robot named | WidowX200 (real); Sawyer arm in a MuJoCo Meta-World simulation |
| Planner | Visual model predictive control with an SV2P video prediction model |
| Code | Official repository linked below |
| Dataset and code licence | Not stated in the sources read |
| Firsthand relationship | None |
11
DVD components, as stated in the paper
| Component | What the sources state |
|---|---|
| Training signal | Triplets of two same-task videos and one different-task video, with a cross-entropy objective |
| Video encoder | Pretrained on Something-Something-V2 and fixed during training |
| Similarity network | Fully connected network, randomly initialized, trained as a binary classifier |
| Batch balance | Each video is a robot demonstration with probability 0.5 |
| Human data | Up to 15 tasks, 853 to 3,170 videos each, from Something-Something-V2 |
| Robot data | 120 simulated demos of 3 tasks; 80 real demos of each of 2 tasks |
| Supervision | Visual input only; task labels for all videos |
12
DVD beside related entries on the site
| Site entry | What the entry states | How DVD differs (sources read) |
|---|---|---|
| VIP | Pre-trains a visual representation on Ego4D human video; embedding distance to a goal image gives a robot reward | DVD learns a reward from labelled video pairs and conditions on a human demo video, not a goal image |
| LIV | Learns a vision-language representation and reward from text-annotated EPIC-KITCHENS video | DVD uses task labels and a small robot dataset, with no language input |
| R3M | Pre-trains a frozen visual representation on Ego4D human video for policy learning | DVD outputs a task-similarity reward, not a perception module |
| WHIRL | Turns one third-person human video into a robot policy, improved by real-world trials scored by an agent-agnostic video comparison | DVD trains once on many human videos and plans with a video prediction model |
| Zero-shot manipulation from passive human videos | Predicts human hand trajectories from a scene image and maps them to a robot | DVD does not predict motion; it scores how similar two videos are |
13
What DVD consumes beside what the Firsthand site states
| Aspect | DVD (sources read) | What the Firsthand site states |
|---|---|---|
| Human video | Existing crowd-sourced Something-Something-V2 videos with task labels | Egocentric capture, custom collection to a written spec |
| Depth | Not a stated input; RGB images | Depth derived from a calibrated stereo rig |
| Hand pose | Not used | 3D hand pose at 21 joints per hand |
| Language | Task labels per video | One task instruction per episode |
| Robot-side data | Small set of simulated and real robot demos | Robot trajectories are not Firsthand deliverables |
| Reward | A learned similarity reward | Reward functions are not Firsthand deliverables |
14
Frequently asked questions
- What is DVD?
- Per the paper (arXiv 2103.16817), DVD, the Domain-agnostic Video Discriminator, learns multitask robot reward functions by classifying whether two videos perform the same task. It is third-party research.
- How does DVD turn human videos into a reward?
- It conditions on a human demonstration video and scores how functionally similar the robot’s behaviour is to it. The score is the reward, and the authors use it with visual model predictive control.
- What data does DVD train on?
- Human videos from Something-Something-V2 plus a small set of robot demos, all with task labels. The dataset licence is Not stated in the sources read.
- What did the authors report for DVD?
- They report a 15 to 20 percent absolute gain in downstream success from diverse human videos, and on a real WidowX200 over 65 to 70 percent success against at most 40 percent for robot-only. Firsthand has not verified these figures.
- Was DVD trained on Firsthand data?
- No. Firsthand is not affiliated with the authors and no named model was trained on Firsthand data. Reward functions, robot trajectories and reward labels are not Firsthand deliverables.
- How can I request a custom egocentric dataset for reward learning research?
- Request a custom dataset with your tasks and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep.
Related terms
- VIP (Value-Implicit Pre-training)
- LIV (Language-Image Representations and Rewards)
- R3M
- WHIRL (Human-to-Robot Imitation in the Wild)
- Zero-Shot Robot Manipulation from Passive Human Videos
- Embodiment gap
- AVID (Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos)
- HuDOR (Bridging the Human to Robot Dexterity Gap through Object-Oriented Rewards)
- Learning Reward Functions for Robotic Manipulation by Observing Humans
- XIRL (Cross-embodiment Inverse Reinforcement Learning)
- GraphIRL (Graph Inverse Reinforcement Learning from Diverse Videos)
Sources
- Learning Generalizable Robotic Reward Functions from “In-The-Wild” Human Videos (arXiv 2103.16817)
- DVD project page (official, linked from the paper)
- DVD official code repository (GitHub)
- The “something something” video database for learning and evaluating visual common sense (arXiv 1706.04261)
- Stochastic Variational Video Prediction, SV2P (arXiv 1710.11252)