Now recruiting capture teams worldwide. Get paid to record first-person video of real work.

Apply to capture

Glossary

What Is Learning by Watching (Physical Imitation of Manipulation Skills from Human Videos)?

LbW learns robot manipulation from one human video via unsupervised translation, keypoints and RL. Third-party; Firsthand is not affiliated.

Last reviewed: November 2026

Short answer

Learning by Watching (LbW) is a framework for physical imitation from human videos, described in arXiv 2101.07241. From one human video, it learns unsupervised human-to-robot translation, detects keypoints on the translated robot video, and uses them to compute a reward for reinforcement learning. Firsthand is not affiliated.

01

What is Learning by Watching, and what problem does it address?

LbW is named in arXiv 2101.07241 by Haoyu Xiong, Quanzhou Li, Yun-Chun Chen, Homanga Bharadhwaj, Samarth Sinha and Animesh Garg (University of Toronto and Vector Institute, Tianjin University, Nvidia). The authors note that expert demonstrations with actions are expensive to collect and that human arms differ in morphology from robot arms. LbW needs only a human video, with no human actions. A venue is Not stated in the sources read.

02

Where does Firsthand fit next to Learning by Watching?

LbW is third-party research. Firsthand is not affiliated with the authors, and no named model, LbW included, was trained on Firsthand data. Firsthand does not claim to use MUNIT, Transporter, Soft Actor-Critic, OpenAI gym or meta-world. The site states custom egocentric collection to a written spec, fast and on demand: first-person human video with calibrated stereo depth, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, in RLDS, LeRobot format or WebDataset, with one task instruction per episode. Translated robot videos, keypoints, reward functions, robot data, trained policies and simulation environments are not Firsthand deliverables.

03

How does keypoint-based human-to-robot video translation work?

Per the paper, the perception module uses MUNIT for unsupervised image-to-image translation, combining each human frame’s content code with a style code sampled from the robot domain. The authors note that translation alone can neglect salient details and leave visual artifacts.

Transporter then detects keypoints on each translated frame without ground-truth keypoint annotations, capturing, the authors say, the robot arm pose and the object location.

04

How does LbW turn keypoints into a robot policy?

The keypoints of the translated demonstration form a trajectory. The reward combines the distance from the current keypoints to the closest demonstration keypoints with a first-order difference term, and a policy trained with Soft Actor-Critic maps keypoints to actions.

05

What data did the LbW authors use?

Policy learning uses one human video per task. The perception module also trains on human videos of random hand motion and on robot videos from a random policy; robot expert videos are not used. Experiments ran in simulation (Fetch-Robot manipulation in OpenAI gym and meta-world) on 84-by-84 RGB observations. No public human video dataset is named, and any licence is Not stated in the sources read.

06

What did the authors report, and what are the limits?

The authors report success rates over 10 test episodes per task in the second table; Firsthand has not verified them. They state limits: translation-based methods need similar human and robot arm poses, and learning from one video limits generalization to new scenes.

07

How does Learning by Watching compare with related entries on this site?

The third table sets LbW beside four site entries. LbW keeps AVID-style translation but rewards detected keypoints instead of stage classifiers.

08

What would an LbW-style approach need from custom collection?

Per the paper, it needs human video per task, random human and robot video, a simulator or robot and a training stack, all the buyer’s own. The paper does not describe egocentric capture, so a buyer should judge whether a capture viewpoint suits their setup. Firsthand covers the human video side only.

Custom collection is on demand: confirm it on a Spec Pilot, with a median of 48 hours to a first validated batch where coverage is deep.

09

Learning by Watching at a glance, as stated in the paper

Learning by Watching at a glance, as stated in the paper
ItemWhat the paper states
PaperarXiv 2101.07241
Method nameLearning by Watching (LbW)
InputOne human demonstration video; no human actions
TranslationMUNIT, unsupervised
KeypointsTransporter detector, unsupervised
Policy learningSoft Actor-Critic on keypoints
Environments namedOpenAI gym Fetch-Robot; meta-world
TasksReaching, pushing, sliding, coffee making, drawer closing
Venue, licenceNot stated in the sources read
Firsthand relationshipNone

11

Success rates reported by the LbW authors (10 test episodes per task; not verified by Firsthand)

Success rates reported by the LbW authors (10 test episodes per task; not verified by Firsthand)
Method (demonstrations used)ReachingPushingSlidingDrawer closingCoffee making
Classifier reward (35 robot videos)100%100%30%70%50%
AVID-m (15 human videos)100%60%0%50%40%
LbW (1 human video)100%100%80%80%70%

12

Learning by Watching beside related entries on the site

Learning by Watching beside related entries on the site
Site entryWhat the entry statesHow LbW differs (sources read)
AVIDCycleGAN image translation, then per-stage model-based reinforcement learningAdds unsupervised keypoints and a keypoint reward; the paper compares against its own AVID reproduction
PhantomHand pose estimation and visual editing, then policy training with no robot dataNo hand pose estimation or rendered robot; policy still learns by reinforcement learning
MasqueradeEdits egocentric videos into robotized demonstrations with a rendered robotUses learned translation; paper does not state egocentric capture
Point PolicyTurns hand poses into robot poses and describes objects with key pointsDetects keypoints without supervision on translated robot frames

13

Frequently asked questions

What is Learning by Watching (LbW)?
Per arXiv 2101.07241, LbW is a framework for policy learning by imitation from a single video: translate the human video to the robot domain, detect keypoints, then learn with a keypoint reward. It is third-party research.
What does physical imitation from human videos mean?
A robot learns to reproduce a manipulation skill shown in human video. The authors state that human actions are not given.
What is keypoint-based human-to-robot video translation?
LbW translates human frames into robot-domain frames, then an unsupervised detector extracts keypoints used for the reward and as policy input.
How many human videos does LbW need?
The authors state that policy learning uses one human expert video per task, plus extra random human and robot video for the perception module.
What did the authors report for LbW?
They report 100%, 100%, 80%, 80% and 70% success on reaching, pushing, sliding, drawer closing and coffee making, in simulation. Firsthand has not verified these.
Was LbW trained on Firsthand data?
No. Firsthand is not affiliated with the authors and no named model was trained on Firsthand data.
How can I request a fast, on-demand custom egocentric dataset for this research?
Request a custom dataset with your tasks and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep.