Glossary
What Is Learning by Watching (Physical Imitation of Manipulation Skills from Human Videos)?
Last reviewed: November 2026
Short answer
Learning by Watching (LbW) is a framework for physical imitation from human videos, described in arXiv 2101.07241. From one human video, it learns unsupervised human-to-robot translation, detects keypoints on the translated robot video, and uses them to compute a reward for reinforcement learning. Firsthand is not affiliated.
01
What is Learning by Watching, and what problem does it address?
LbW is named in arXiv 2101.07241 by Haoyu Xiong, Quanzhou Li, Yun-Chun Chen, Homanga Bharadhwaj, Samarth Sinha and Animesh Garg (University of Toronto and Vector Institute, Tianjin University, Nvidia). The authors note that expert demonstrations with actions are expensive to collect and that human arms differ in morphology from robot arms. LbW needs only a human video, with no human actions. A venue is Not stated in the sources read.
02
Where does Firsthand fit next to Learning by Watching?
LbW is third-party research. Firsthand is not affiliated with the authors, and no named model, LbW included, was trained on Firsthand data. Firsthand does not claim to use MUNIT, Transporter, Soft Actor-Critic, OpenAI gym or meta-world. The site states custom egocentric collection to a written spec, fast and on demand: first-person human video with calibrated stereo depth, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, in RLDS, LeRobot format or WebDataset, with one task instruction per episode. Translated robot videos, keypoints, reward functions, robot data, trained policies and simulation environments are not Firsthand deliverables.
03
How does keypoint-based human-to-robot video translation work?
Per the paper, the perception module uses MUNIT for unsupervised image-to-image translation, combining each human frame’s content code with a style code sampled from the robot domain. The authors note that translation alone can neglect salient details and leave visual artifacts.
Transporter then detects keypoints on each translated frame without ground-truth keypoint annotations, capturing, the authors say, the robot arm pose and the object location.
04
How does LbW turn keypoints into a robot policy?
The keypoints of the translated demonstration form a trajectory. The reward combines the distance from the current keypoints to the closest demonstration keypoints with a first-order difference term, and a policy trained with Soft Actor-Critic maps keypoints to actions.
05
What data did the LbW authors use?
Policy learning uses one human video per task. The perception module also trains on human videos of random hand motion and on robot videos from a random policy; robot expert videos are not used. Experiments ran in simulation (Fetch-Robot manipulation in OpenAI gym and meta-world) on 84-by-84 RGB observations. No public human video dataset is named, and any licence is Not stated in the sources read.
06
What did the authors report, and what are the limits?
The authors report success rates over 10 test episodes per task in the second table; Firsthand has not verified them. They state limits: translation-based methods need similar human and robot arm poses, and learning from one video limits generalization to new scenes.
07
How does Learning by Watching compare with related entries on this site?
The third table sets LbW beside four site entries. LbW keeps AVID-style translation but rewards detected keypoints instead of stage classifiers.
08
What would an LbW-style approach need from custom collection?
Per the paper, it needs human video per task, random human and robot video, a simulator or robot and a training stack, all the buyer’s own. The paper does not describe egocentric capture, so a buyer should judge whether a capture viewpoint suits their setup. Firsthand covers the human video side only.
Custom collection is on demand: confirm it on a Spec Pilot, with a median of 48 hours to a first validated batch where coverage is deep.
09
Learning by Watching at a glance, as stated in the paper
| Item | What the paper states |
|---|---|
| Paper | arXiv 2101.07241 |
| Method name | Learning by Watching (LbW) |
| Input | One human demonstration video; no human actions |
| Translation | MUNIT, unsupervised |
| Keypoints | Transporter detector, unsupervised |
| Policy learning | Soft Actor-Critic on keypoints |
| Environments named | OpenAI gym Fetch-Robot; meta-world |
| Tasks | Reaching, pushing, sliding, coffee making, drawer closing |
| Venue, licence | Not stated in the sources read |
| Firsthand relationship | None |
11
Success rates reported by the LbW authors (10 test episodes per task; not verified by Firsthand)
| Method (demonstrations used) | Reaching | Pushing | Sliding | Drawer closing | Coffee making |
|---|---|---|---|---|---|
| Classifier reward (35 robot videos) | 100% | 100% | 30% | 70% | 50% |
| AVID-m (15 human videos) | 100% | 60% | 0% | 50% | 40% |
| LbW (1 human video) | 100% | 100% | 80% | 80% | 70% |
12
Learning by Watching beside related entries on the site
| Site entry | What the entry states | How LbW differs (sources read) |
|---|---|---|
| AVID | CycleGAN image translation, then per-stage model-based reinforcement learning | Adds unsupervised keypoints and a keypoint reward; the paper compares against its own AVID reproduction |
| Phantom | Hand pose estimation and visual editing, then policy training with no robot data | No hand pose estimation or rendered robot; policy still learns by reinforcement learning |
| Masquerade | Edits egocentric videos into robotized demonstrations with a rendered robot | Uses learned translation; paper does not state egocentric capture |
| Point Policy | Turns hand poses into robot poses and describes objects with key points | Detects keypoints without supervision on translated robot frames |
13
Frequently asked questions
- What is Learning by Watching (LbW)?
- Per arXiv 2101.07241, LbW is a framework for policy learning by imitation from a single video: translate the human video to the robot domain, detect keypoints, then learn with a keypoint reward. It is third-party research.
- What does physical imitation from human videos mean?
- A robot learns to reproduce a manipulation skill shown in human video. The authors state that human actions are not given.
- What is keypoint-based human-to-robot video translation?
- LbW translates human frames into robot-domain frames, then an unsupervised detector extracts keypoints used for the reward and as policy input.
- How many human videos does LbW need?
- The authors state that policy learning uses one human expert video per task, plus extra random human and robot video for the perception module.
- What did the authors report for LbW?
- They report 100%, 100%, 80%, 80% and 70% success on reaching, pushing, sliding, drawer closing and coffee making, in simulation. Firsthand has not verified these.
- Was LbW trained on Firsthand data?
- No. Firsthand is not affiliated with the authors and no named model was trained on Firsthand data.
- How can I request a fast, on-demand custom egocentric dataset for this research?
- Request a custom dataset with your tasks and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep.
Related terms
Sources
- Learning by Watching: Physical Imitation of Manipulation Skills from Human Videos (arXiv 2101.07241)
- Multimodal Unsupervised Image-to-Image Translation (MUNIT, arXiv 1804.04732)
- Unsupervised Learning of Object Keypoints for Perception and Control (Transporter, arXiv 1906.11883)
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (arXiv 1801.01290)
- AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos (arXiv 1912.04443)