Now recruiting capture teams worldwide. Get paid to record first-person video of real work.

Apply to capture

Glossary

What Is DVD (Learning Generalizable Robotic Reward Functions from In-the-Wild Human Videos)?

DVD is a published method that learns a robot reward by comparing a robot video with a human demo video. Third-party; Firsthand is not affiliated.

Last reviewed: November 2026

Short answer

DVD, the Domain-agnostic Video Discriminator, is the method in “Learning Generalizable Robotic Reward Functions from ‘In-The-Wild’ Human Videos” (arXiv 2103.16817, RSS 2021). It trains a discriminator to judge whether two videos show the same task, using a few robot demos plus many human videos, and the score serves as a robot reward. Firsthand is not affiliated.

01

What is DVD, and what problem does it address?

DVD is named in Learning Generalizable Robotic Reward Functions from “In-The-Wild” Human Videos (arXiv 2103.16817) by Annie S. Chen, Suraj Nair and Chelsea Finn of Stanford University, published at Robotics: Science and Systems. The authors want a robot to acquire a measure of task success, a reward, that generalizes across environments, tasks and objects while depending only on on-board sensor observations such as RGB images. They note that collecting high-quality robot interaction data at scale remains an open challenge, whereas in-the-wild human videos, for example from YouTube, show people doing many tasks in diverse settings.

02

Where does Firsthand fit next to DVD?

Firsthand is not affiliated with the DVD authors. No named model, DVD included, was trained on Firsthand data, and Firsthand does not claim to use any hardware or software named in the sources: they name the WidowX200 robot, a Sawyer arm in a MuJoCo simulation built on Meta-World, and the Something-Something-V2 dataset, which Firsthand does not supply. Firsthand captures real human egocentric video, custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license, with one task instruction per episode. Reward functions, reward or value labels, robot trajectories, action labels, simulation environments, digital twins, co-training mixtures, pretrained representations, embeddings, synthetic data and language annotations beyond that instruction are not Firsthand deliverables.

03

How does DVD learn a reward from videos?

Per the paper, DVD trains a classifier that takes two videos and predicts whether they complete the same task. Training batches are triplets: two videos labelled with the same task and one labelled with a different task, drawn from a mix of human and robot videos. The authors say this forces the model to learn the functional similarity between videos from very different visual domains.

The reward is a similarity score between a human demonstration video, which specifies the task, and the robot’s own behaviour. A pretrained video encoder, fixed during training, feeds a fully connected network that outputs the score. The authors state that the reward is inferred only from visual observations and assumes no actions or low-dimensional states from the human or robot data.

04

What video does DVD train on?

The human side is the existing Something-Something-V2 dataset, which the paper describes as 220,837 videos in 174 classes. Depending on the experiment the authors use up to 15 human tasks, each with 853 to 3,170 videos, and place no constraint on viewpoint, background or quality. The robot side is small: 120 simulated demonstrations of 3 tasks, and 80 demonstrations of each of 2 tasks on the real robot, which the authors say were collected by scripted or model-predictive routines.

The dataset licence is Not stated in the sources read.

05

How is the learned reward used to choose actions?

The authors pair DVD with visual model predictive control. An action-conditioned video prediction model, trained with SV2P, imagines the future for many sampled action sequences. The cross-entropy method then picks the sequence whose predicted video DVD scores as most similar to the human demo.

06

What did the authors report, and what are the limits?

The authors report that training on diverse human videos gives a 15 to 20 percent absolute improvement in downstream task success over using robot data alone, including for tasks unrelated to the target. On the real WidowX200 they report 20 trials per task in an unseen environment, with DVD trained on 6 human tasks succeeding over 65 to 70 percent of the time against at most 40 percent for robot-only. These are the authors’ figures; Firsthand has not verified them.

The paper states limits: it learns a reward, not a policy or a dynamics model; it needs some robot demos and task labels for all videos; and it is tested only on coarse tasks that do not need fine-grained manipulation. The authors report that success falls 6 percent with 20 robot demos per task and 13 percent with none.

07

How does DVD compare with related entries on this site?

The comparison table sets DVD beside five site entries. The main difference is that DVD learns a task-conditioned reward from labelled video pairs, where the entries it is compared with pretrain a representation, imitate one video, or predict hand motion.

08

What would a DVD-style approach need from custom collection?

Per the sources read, it needs labelled human videos of the target tasks, a small set of robot demonstrations, a video encoder, a video prediction model and a robot or simulator, all the buyer’s own. Firsthand states one task instruction per episode and no other labels, so any per-video task-class scheme is the buyer’s own. The paper does not state a need for stereo depth, hand pose or IMUs.

No off-the-shelf dataset for this is stated here, so this is custom collection: request on-demand collection and confirm it on a Spec Pilot, with a median of 48 hours to a first validated batch where coverage is deep and three to four weeks where it is thin.

09

DVD at a glance, as stated in the paper, project page and code repository

DVD at a glance, as stated in the paper, project page and code repository
ItemWhat the sources state
PaperLearning Generalizable Robotic Reward Functions from “In-The-Wild” Human Videos (arXiv 2103.16817)
Method nameDomain-agnostic Video Discriminator (DVD)
AuthorsAnnie S. Chen, Suraj Nair, Chelsea Finn
InstitutionStanford University
VenueRobotics: Science and Systems, 2021, per the project page
Core ideaTrain a discriminator to classify whether two videos perform the same task; use its score as a reward
Human inputSomething-Something-V2 videos with task labels
Robot namedWidowX200 (real); Sawyer arm in a MuJoCo Meta-World simulation
PlannerVisual model predictive control with an SV2P video prediction model
CodeOfficial repository linked below
Dataset and code licenceNot stated in the sources read
Firsthand relationshipNone

11

DVD components, as stated in the paper

DVD components, as stated in the paper
ComponentWhat the sources state
Training signalTriplets of two same-task videos and one different-task video, with a cross-entropy objective
Video encoderPretrained on Something-Something-V2 and fixed during training
Similarity networkFully connected network, randomly initialized, trained as a binary classifier
Batch balanceEach video is a robot demonstration with probability 0.5
Human dataUp to 15 tasks, 853 to 3,170 videos each, from Something-Something-V2
Robot data120 simulated demos of 3 tasks; 80 real demos of each of 2 tasks
SupervisionVisual input only; task labels for all videos

12

DVD beside related entries on the site

DVD beside related entries on the site
Site entryWhat the entry statesHow DVD differs (sources read)
VIPPre-trains a visual representation on Ego4D human video; embedding distance to a goal image gives a robot rewardDVD learns a reward from labelled video pairs and conditions on a human demo video, not a goal image
LIVLearns a vision-language representation and reward from text-annotated EPIC-KITCHENS videoDVD uses task labels and a small robot dataset, with no language input
R3MPre-trains a frozen visual representation on Ego4D human video for policy learningDVD outputs a task-similarity reward, not a perception module
WHIRLTurns one third-person human video into a robot policy, improved by real-world trials scored by an agent-agnostic video comparisonDVD trains once on many human videos and plans with a video prediction model
Zero-shot manipulation from passive human videosPredicts human hand trajectories from a scene image and maps them to a robotDVD does not predict motion; it scores how similar two videos are

13

What DVD consumes beside what the Firsthand site states

What DVD consumes beside what the Firsthand site states
AspectDVD (sources read)What the Firsthand site states
Human videoExisting crowd-sourced Something-Something-V2 videos with task labelsEgocentric capture, custom collection to a written spec
DepthNot a stated input; RGB imagesDepth derived from a calibrated stereo rig
Hand poseNot used3D hand pose at 21 joints per hand
LanguageTask labels per videoOne task instruction per episode
Robot-side dataSmall set of simulated and real robot demosRobot trajectories are not Firsthand deliverables
RewardA learned similarity rewardReward functions are not Firsthand deliverables

14

Frequently asked questions

What is DVD?
Per the paper (arXiv 2103.16817), DVD, the Domain-agnostic Video Discriminator, learns multitask robot reward functions by classifying whether two videos perform the same task. It is third-party research.
How does DVD turn human videos into a reward?
It conditions on a human demonstration video and scores how functionally similar the robot’s behaviour is to it. The score is the reward, and the authors use it with visual model predictive control.
What data does DVD train on?
Human videos from Something-Something-V2 plus a small set of robot demos, all with task labels. The dataset licence is Not stated in the sources read.
What did the authors report for DVD?
They report a 15 to 20 percent absolute gain in downstream success from diverse human videos, and on a real WidowX200 over 65 to 70 percent success against at most 40 percent for robot-only. Firsthand has not verified these figures.
Was DVD trained on Firsthand data?
No. Firsthand is not affiliated with the authors and no named model was trained on Firsthand data. Reward functions, robot trajectories and reward labels are not Firsthand deliverables.
How can I request a custom egocentric dataset for reward learning research?
Request a custom dataset with your tasks and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep.