Now recruiting capture teams worldwide. Get paid to record first-person video of real work.

Apply to capture

Glossary

What Is AVID (Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos)?

AVID translates human demonstration images into robot images with CycleGAN, then learns each task stage with model-based RL. Third-party; Firsthand is not affiliated.

Last reviewed: November 2026

Short answer

AVID, automated visual instruction-following with demonstrations, is the method in “AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos” (arXiv 1912.04443, RSS). It uses CycleGAN to translate human demonstration images into robot images, then a robot learns each stage with model-based reinforcement learning and classifier rewards. Firsthand is not affiliated.

01

What is AVID, and what problem does it address?

AVID is named in AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos (arXiv 1912.04443) by Laura Smith, Nikita Dhawan, Marvin Zhang, Pieter Abbeel and Sergey Levine of Berkeley Artificial Intelligence Research; the project page lists Robotics: Science and Systems, 2020. The authors note that long-horizon real-world tasks need ways to reduce the human burden of defining the task and scaffolding learning, and that a central challenge in imitating human videos is the difference in appearance between human and robot, which typically requires manual correspondence. They replace that correspondence with learned pixel-level image translation. See also the site’s embodiment gap, Phantom, Masquerade and DVD entries.

02

Where does Firsthand fit next to AVID?

Firsthand is not affiliated with the AVID authors. No named model, AVID included, was trained on Firsthand data, and Firsthand does not claim to use any hardware or software named in the paper: a 7 DoF Sawyer robotic manipulator, CycleGAN and a latent variable model with model-predictive planning. Firsthand captures real human egocentric video, custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license, with one task instruction per episode. Translated or synthetic robot images or video, instruction images, success classifiers, reward functions, latent models, robot trajectories, retargeted actions, action labels, key-press feedback, pretrained representations, embeddings, simulation environments, synthetic data and language annotations beyond that instruction are not Firsthand deliverables.

03

How does AVID translate human videos into robot images?

Per the paper, AVID treats human-to-robot translation as an unsupervised image-to-image problem, mapping images from a human domain to a robot domain without paired training data. It uses CycleGAN, which learns two mappings, one in each direction, plus a cycle-consistency loss. The authors note CycleGAN does not use the temporal information in a video, so they translate frame by frame.

The authors state that the approach assumes a bijective mapping that can be uncovered without supervision, and that in their experiments this shows up as similarities in morphology and perspective between human and robot images. They also report that translated images contain artifacts, such as a displaced robot gripper and a deformed cup.

04

How does AVID learn a multi-stage task?

A person manually picks the time steps in the translated video that mark the end of each stage. Those frames become instruction images. For each stage the authors train a success classifier on the instruction images as positives and all other robot images as negatives, working on a learned latent state, and use its log probability as the reward.

The robot then plans in that latent space with model-predictive control using the cross-entropy method. If a stage is not reached it switches to the previous stage’s classifier to reset, then retries. When the classifier signals success, the human confirms with a key press, and a failure triggers extra exploration and classifier retraining. The authors say this stage-wise loop avoids compounding errors from learning the whole task at once.

05

What data did the AVID authors use?

The authors filmed their own human demonstration videos and random robot trajectories. For coffee making they report 30 human demonstrations (900 images) plus 500 images of random human data, taking about 20 minutes; for cup retrieval, 20 demonstrations (600 images) plus 300 random human images. The robot domain data is the robot executing randomly sampled actions, with a person changing the setting between runs. Observations are 64-by-64-by-3 RGB images on a 7 DoF Sawyer robotic manipulator.

No public human video dataset is named as a source. Any dataset or code licence is Not stated in the sources read. The paper links a project website with a supplementary video, which the authors say shows the learning process and final performance.

06

What did the authors report, and what are the limits?

The authors report that AVID succeeded 8 times out of 10 on coffee making and 7 out of 10 on cup retrieval, above the other methods that learn from human demonstrations, and used 131 and 126 human key presses respectively, fewer than 150 each. They report training took about one hour per task. These are the authors’ figures; Firsthand has not verified them.

Stated limits: stages are specified by hand, a separate CycleGAN is trained per task, learning needs human feedback and a few manual resets, and only two tasks were evaluated. The authors list training one translation model across tasks as future work.

07

How does AVID compare with related entries on this site?

The table sets AVID beside four site entries. Unlike Phantom and Masquerade, AVID does not render a robot over the video; it learns the translation, then still has the robot practice with reinforcement learning.

08

What would an AVID-style approach need from custom collection?

Per the paper, it needs human demonstration videos plus some random human video, random robot data in matching settings, a robot, and a training stack, all the buyer’s own. The paper does not describe egocentric capture and assumes similar morphology and perspective between human and robot images, so a buyer would need to judge whether a capture viewpoint suits their robot images. Firsthand covers the human video side only.

This is custom collection: request on-demand collection and confirm it on a Spec Pilot, with a median of 48 hours to a first validated batch where coverage is deep and three to four weeks where it is thin.

09

AVID at a glance, as stated in the paper and project page

AVID at a glance, as stated in the paper and project page
ItemWhat the sources state
PaperAVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos (arXiv 1912.04443)
AuthorsLaura Smith, Nikita Dhawan, Marvin Zhang, Pieter Abbeel, Sergey Levine
VenueRobotics: Science and Systems (2020, per the project page)
Method nameAutomated visual instruction-following with demonstrations (AVID)
TranslationCycleGAN, unpaired image-to-image translation, frame by frame
RewardPer-stage success classifiers on a learned latent state
PlanningLatent-space model-predictive control with the cross-entropy method
Robot named7 DoF Sawyer robotic manipulator
TasksOperating a coffee machine; retrieving a cup from a drawer
Dataset and code licenceNot stated in the sources read
Firsthand relationshipNone

11

AVID beside related entries on the site

AVID beside related entries on the site
Site entryWhat the entry statesHow AVID differs (sources read)
PhantomTurns human video into robot observation-action pairs with hand pose estimation and visual data editing, then trains a policy with no robot dataAVID learns an unpaired image translation instead of estimating hand pose and rendering a robot, and its robot still practises with reinforcement learning
MasqueradeEdits egocentric human videos into robotized demonstrations by estimating hand poses, inpainting arms and overlaying a rendered robotAVID does not estimate hand poses or inpaint, and the paper does not state egocentric capture
DVDTrains a discriminator that judges whether two videos show the same task, and uses the score as a robot rewardAVID builds its reward from per-stage classifiers on translated instruction images
AR2-D2Collects robot demonstrations with a mobile AR app, with no real robotAVID starts from ordinary human videos and has the robot learn by practice

12

What AVID consumes beside what the Firsthand site states

What AVID consumes beside what the Firsthand site states
AspectAVID (sources read)What the Firsthand site states
Human videoHuman demonstration videos plus a small amount of random human video, as image observationsEgocentric capture, custom collection to a written spec
Robot-side dataRandom robot trajectories in a few settings, plus online robot dataRobot trajectories are not Firsthand deliverables
DepthNot used; observations are RGB imagesDepth derived from a calibrated stereo rig
Hand poseNot used by the method3D hand pose at 21 joints per hand
LanguageNone; the task is set by demonstration videosOne task instruction per episode
Human feedbackKey presses and a few manual resetsNot a Firsthand deliverable

13

Frequently asked questions

What is AVID?
Per the paper (arXiv 1912.04443), AVID is automated visual instruction-following with demonstrations. It translates human demonstration images into robot images with CycleGAN, then a robot learns each stage with model-based reinforcement learning. It is third-party research.
What does pixel-level translation of human videos mean?
It means converting images of a human doing a task into images of a robot in the same scene, using learned image-to-image translation instead of manually matching human and robot keypoints. AVID uses CycleGAN and translates frame by frame.
How does AVID learn multi-stage robot tasks from human demonstrations?
A person picks a few stage-completion frames from the translated video. These train one success classifier per stage, which acts as the reward, and the robot practises each stage with learned resets and human key presses.
What did the authors report for AVID?
They report 8 successes in 10 trials on coffee making and 7 in 10 on cup retrieval, with fewer than 150 key presses per task. Firsthand has not verified these figures.
Does AVID learn from human video alone?
No. It also uses random robot data to train the translation and the latent model, and the robot then practises on the real task with human key presses and a few manual resets.
Was AVID trained on Firsthand data?
No. Firsthand is not affiliated with the authors and no named model was trained on Firsthand data. Translated robot images, classifiers and robot trajectories are not Firsthand deliverables.
How can I request a fast, on-demand custom egocentric dataset for this research?
Request a custom dataset with your tasks and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep.