Glossary
What Is AVID (Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos)?
Last reviewed: November 2026
Short answer
AVID, automated visual instruction-following with demonstrations, is the method in “AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos” (arXiv 1912.04443, RSS). It uses CycleGAN to translate human demonstration images into robot images, then a robot learns each stage with model-based reinforcement learning and classifier rewards. Firsthand is not affiliated.
01
What is AVID, and what problem does it address?
AVID is named in AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos (arXiv 1912.04443) by Laura Smith, Nikita Dhawan, Marvin Zhang, Pieter Abbeel and Sergey Levine of Berkeley Artificial Intelligence Research; the project page lists Robotics: Science and Systems, 2020. The authors note that long-horizon real-world tasks need ways to reduce the human burden of defining the task and scaffolding learning, and that a central challenge in imitating human videos is the difference in appearance between human and robot, which typically requires manual correspondence. They replace that correspondence with learned pixel-level image translation. See also the site’s embodiment gap, Phantom, Masquerade and DVD entries.
02
Where does Firsthand fit next to AVID?
Firsthand is not affiliated with the AVID authors. No named model, AVID included, was trained on Firsthand data, and Firsthand does not claim to use any hardware or software named in the paper: a 7 DoF Sawyer robotic manipulator, CycleGAN and a latent variable model with model-predictive planning. Firsthand captures real human egocentric video, custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license, with one task instruction per episode. Translated or synthetic robot images or video, instruction images, success classifiers, reward functions, latent models, robot trajectories, retargeted actions, action labels, key-press feedback, pretrained representations, embeddings, simulation environments, synthetic data and language annotations beyond that instruction are not Firsthand deliverables.
03
How does AVID translate human videos into robot images?
Per the paper, AVID treats human-to-robot translation as an unsupervised image-to-image problem, mapping images from a human domain to a robot domain without paired training data. It uses CycleGAN, which learns two mappings, one in each direction, plus a cycle-consistency loss. The authors note CycleGAN does not use the temporal information in a video, so they translate frame by frame.
The authors state that the approach assumes a bijective mapping that can be uncovered without supervision, and that in their experiments this shows up as similarities in morphology and perspective between human and robot images. They also report that translated images contain artifacts, such as a displaced robot gripper and a deformed cup.
04
How does AVID learn a multi-stage task?
A person manually picks the time steps in the translated video that mark the end of each stage. Those frames become instruction images. For each stage the authors train a success classifier on the instruction images as positives and all other robot images as negatives, working on a learned latent state, and use its log probability as the reward.
The robot then plans in that latent space with model-predictive control using the cross-entropy method. If a stage is not reached it switches to the previous stage’s classifier to reset, then retries. When the classifier signals success, the human confirms with a key press, and a failure triggers extra exploration and classifier retraining. The authors say this stage-wise loop avoids compounding errors from learning the whole task at once.
05
What data did the AVID authors use?
The authors filmed their own human demonstration videos and random robot trajectories. For coffee making they report 30 human demonstrations (900 images) plus 500 images of random human data, taking about 20 minutes; for cup retrieval, 20 demonstrations (600 images) plus 300 random human images. The robot domain data is the robot executing randomly sampled actions, with a person changing the setting between runs. Observations are 64-by-64-by-3 RGB images on a 7 DoF Sawyer robotic manipulator.
No public human video dataset is named as a source. Any dataset or code licence is Not stated in the sources read. The paper links a project website with a supplementary video, which the authors say shows the learning process and final performance.
06
What did the authors report, and what are the limits?
The authors report that AVID succeeded 8 times out of 10 on coffee making and 7 out of 10 on cup retrieval, above the other methods that learn from human demonstrations, and used 131 and 126 human key presses respectively, fewer than 150 each. They report training took about one hour per task. These are the authors’ figures; Firsthand has not verified them.
Stated limits: stages are specified by hand, a separate CycleGAN is trained per task, learning needs human feedback and a few manual resets, and only two tasks were evaluated. The authors list training one translation model across tasks as future work.
07
How does AVID compare with related entries on this site?
The table sets AVID beside four site entries. Unlike Phantom and Masquerade, AVID does not render a robot over the video; it learns the translation, then still has the robot practice with reinforcement learning.
08
What would an AVID-style approach need from custom collection?
Per the paper, it needs human demonstration videos plus some random human video, random robot data in matching settings, a robot, and a training stack, all the buyer’s own. The paper does not describe egocentric capture and assumes similar morphology and perspective between human and robot images, so a buyer would need to judge whether a capture viewpoint suits their robot images. Firsthand covers the human video side only.
This is custom collection: request on-demand collection and confirm it on a Spec Pilot, with a median of 48 hours to a first validated batch where coverage is deep and three to four weeks where it is thin.
09
AVID at a glance, as stated in the paper and project page
| Item | What the sources state |
|---|---|
| Paper | AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos (arXiv 1912.04443) |
| Authors | Laura Smith, Nikita Dhawan, Marvin Zhang, Pieter Abbeel, Sergey Levine |
| Venue | Robotics: Science and Systems (2020, per the project page) |
| Method name | Automated visual instruction-following with demonstrations (AVID) |
| Translation | CycleGAN, unpaired image-to-image translation, frame by frame |
| Reward | Per-stage success classifiers on a learned latent state |
| Planning | Latent-space model-predictive control with the cross-entropy method |
| Robot named | 7 DoF Sawyer robotic manipulator |
| Tasks | Operating a coffee machine; retrieving a cup from a drawer |
| Dataset and code licence | Not stated in the sources read |
| Firsthand relationship | None |
11
AVID beside related entries on the site
| Site entry | What the entry states | How AVID differs (sources read) |
|---|---|---|
| Phantom | Turns human video into robot observation-action pairs with hand pose estimation and visual data editing, then trains a policy with no robot data | AVID learns an unpaired image translation instead of estimating hand pose and rendering a robot, and its robot still practises with reinforcement learning |
| Masquerade | Edits egocentric human videos into robotized demonstrations by estimating hand poses, inpainting arms and overlaying a rendered robot | AVID does not estimate hand poses or inpaint, and the paper does not state egocentric capture |
| DVD | Trains a discriminator that judges whether two videos show the same task, and uses the score as a robot reward | AVID builds its reward from per-stage classifiers on translated instruction images |
| AR2-D2 | Collects robot demonstrations with a mobile AR app, with no real robot | AVID starts from ordinary human videos and has the robot learn by practice |
12
What AVID consumes beside what the Firsthand site states
| Aspect | AVID (sources read) | What the Firsthand site states |
|---|---|---|
| Human video | Human demonstration videos plus a small amount of random human video, as image observations | Egocentric capture, custom collection to a written spec |
| Robot-side data | Random robot trajectories in a few settings, plus online robot data | Robot trajectories are not Firsthand deliverables |
| Depth | Not used; observations are RGB images | Depth derived from a calibrated stereo rig |
| Hand pose | Not used by the method | 3D hand pose at 21 joints per hand |
| Language | None; the task is set by demonstration videos | One task instruction per episode |
| Human feedback | Key presses and a few manual resets | Not a Firsthand deliverable |
13
Frequently asked questions
- What is AVID?
- Per the paper (arXiv 1912.04443), AVID is automated visual instruction-following with demonstrations. It translates human demonstration images into robot images with CycleGAN, then a robot learns each stage with model-based reinforcement learning. It is third-party research.
- What does pixel-level translation of human videos mean?
- It means converting images of a human doing a task into images of a robot in the same scene, using learned image-to-image translation instead of manually matching human and robot keypoints. AVID uses CycleGAN and translates frame by frame.
- How does AVID learn multi-stage robot tasks from human demonstrations?
- A person picks a few stage-completion frames from the translated video. These train one success classifier per stage, which acts as the reward, and the robot practises each stage with learned resets and human key presses.
- What did the authors report for AVID?
- They report 8 successes in 10 trials on coffee making and 7 in 10 on cup retrieval, with fewer than 150 key presses per task. Firsthand has not verified these figures.
- Does AVID learn from human video alone?
- No. It also uses random robot data to train the translation and the latent model, and the robot then practises on the real task with human key presses and a few manual resets.
- Was AVID trained on Firsthand data?
- No. Firsthand is not affiliated with the authors and no named model was trained on Firsthand data. Translated robot images, classifiers and robot trajectories are not Firsthand deliverables.
- How can I request a fast, on-demand custom egocentric dataset for this research?
- Request a custom dataset with your tasks and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep.
Related terms
- Masquerade (Learning from In-the-Wild Human Videos via Data-Editing)
- Phantom (Training Robots Without Robots Using Only Human Videos)
- DVD (Learning Generalizable Robotic Reward Functions from In-the-Wild Human Videos)
- Embodiment gap
- Cross-embodiment learning
- Imitation learning
- Learning by Watching (Physical Imitation of Manipulation Skills from Human Videos)
- Domain-Adaptive Meta-Learning (DAML)
- Imitation from Observation (Context Translation)
Sources
- AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos (arXiv 1912.04443)
- AVID project page (official, linked from the paper)
- Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks (CycleGAN, arXiv 1703.10593)
- Time-Contrastive Networks: Self-Supervised Learning from Video (arXiv 1704.06888)
- Phantom: Training Robots Without Robots Using Only Human Videos (arXiv 2503.00779)