Now recruiting capture teams worldwide. Get paid to record first-person video of real work.

Apply to capture

Glossary

What Is Third-Person Visual Imitation Learning via a Decoupled Hierarchical Controller?

A robot imitates one third-person human video: a high-level module sets sub-goals, a low-level controller acts. Third-party; Firsthand is not affiliated.

Last reviewed: November 2026

Short answer

Third-person visual imitation learning via a decoupled hierarchical controller is the method in arXiv 1911.09676 (NeurIPS 2019). A robot watches a single third-person video of a human and does the same task in its own setting. A high-level module generates first-person sub-goals from that video, and a low-level controller predicts actions to reach them. Third-party research; Firsthand is not affiliated.

01

What is third-person visual imitation learning, and what problem does it address?

The paper, Third-Person Visual Imitation Learning via Decoupled Hierarchical Controller, is by Pratyusha Sharma, Deepak Pathak and Abhinav Gupta. The authors study robot imitation from a human video of a different viewpoint: an agent that manipulates novel objects in unseen scenarios by looking at only a single third-person human demonstration. The agent must understand the intent of the video in its context and also perform the task in its own environment configuration. Their central insight is to enforce this structure during learning by decoupling what to achieve (the intended task) from how to perform it (the controller). See also the site’s imitation learning and embodiment gap entries.

02

Where does Firsthand fit next to this method?

This is third-party research. Firsthand is not affiliated with the authors, and no named model, including this one, was trained on Firsthand data. Firsthand does not claim to use a Baxter robot or any software the paper names. The site states custom egocentric collection, fast and on demand: first-person human video with calibrated stereo depth, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, with one task instruction per episode. Per the site’s egocentric vs third-person comparison, Firsthand does not capture third-person or fixed external-camera footage. Robot demonstrations, teleoperation data, sub-goal images and trained policies are not Firsthand deliverables.

03

How does the decoupled hierarchical controller work?

Per the abstract, the setup is hierarchical. A high-level module learns to generate a series of first-person sub-goals conditioned on the third-person video demonstration. A low-level controller then predicts the actions that achieve those sub-goals. The agent acts from raw image observations, without access to full state information.

04

What does "decoupled" mean in training and at inference?

The project page describes a goal generator and a low-level controller that are trained independently. The goal generator takes two frames of the human video and the observed robot state, and predicts the visual goal state of the robot at a later step. The low-level controller is trained from state, action and next-state triplets. At inference the two models run one after the other in a loop. The first table summarizes these items.

05

What data and robot did the authors use?

The abstract reports results on a real robotic platform, a Baxter, for pouring and placing objects in a box. The abstract and project page name no public human-video dataset. The sizes of the training sets, and the licence of any data, are Not stated in the sources read. The paper also reports figures and comparisons that are not repeated here, because the pages read for this entry do not state them.

06

How does it differ from related methods on this site?

The second table sets this method beside five site entries. It uses only what those entries state and what the sources read state, and it makes no performance comparison.

07

What would this kind of approach need from custom collection?

Per the sources read, the method needs a human video from a third-person viewpoint plus the robot data to train the controller, which the buyer would collect on their own robot. Firsthand covers a different viewpoint: first-person human video, as custom collection at scale through a network of thousands of capture providers, for a consistent, recurring data flow to labs, confirmed on a Spec Pilot. Any use of that video with this method would be the buyer’s own research.

08

Third-person visual imitation via decoupled hierarchical controller, as stated in the sources read

Third-person visual imitation via decoupled hierarchical controller, as stated in the sources read
ItemWhat the sources state
PaperThird-Person Visual Imitation Learning via Decoupled Hierarchical Controller (arXiv 1911.09676)
Authors and venuePratyusha Sharma, Deepak Pathak, Abhinav Gupta; NeurIPS 2019
Input at test timeA single video of a human demonstration from a third-person perspective
High-level moduleA goal generator that produces first-person sub-goals from the third-person video
Low-level controllerPredicts actions to reach each sub-goal; trained from state, action and next-state triplets
ObservationsRaw images, with no access to full state information
Robot and tasksA Baxter; pouring and placing objects in a box
CodeA PyTorch implementation is linked from the official project page
LicencesNot stated in the sources read
Firsthand relationshipNone

10

This method beside related entries on the site

This method beside related entries on the site
Site entryWhat the entry statesHow this method differs (sources read)
WHIRLUses one third-person human video, extracts hand-motion and contact priors, then improves through real-world trialsThe sources read describe a goal generator and a low-level controller, and no trial-based improvement step
AVIDTranslates human demonstration images into robot images with CycleGAN, then learns each stage with model-based reinforcement learningHere a goal generator predicts the robot’s visual goal state; the sources read mention no image translation
DAMLMeta-learns a prior, then learns a task from one human videoThe sources read describe two decoupled models rather than a meta-learned prior
Time-Contrastive NetworksLearns a viewpoint-invariant embedding from unlabeled multi-view videoHere the video conditions a sub-goal generator; the sources read name no embedding objective
Embodiment gapThe mismatch between a human body and a robot’s hardwareThe method addresses a human demonstration seen from a third-person viewpoint, then acts in the robot’s own view

11

Frequently asked questions

What is third-person visual imitation learning via a decoupled hierarchical controller?
Per arXiv 1911.09676, a robot watches one third-person human video, a high-level module generates first-person sub-goals, and a low-level controller predicts actions to reach them. It is third-party research.
What does decoupling mean here?
The authors decouple what to achieve, the intended task, from how to perform it, the controller. The project page says the two models are trained independently and run in a loop at inference.
Does the agent need the full state of the world?
No. The abstract says the agent acts from raw image observations without any access to full state information.
Which robot and tasks does the paper report?
A Baxter, for pouring and placing objects in a box, per the abstract.
Does the method use first-person video?
The input is a third-person human video, and the high-level module generates first-person sub-goals. Firsthand captures first-person human video and does not capture third-person footage.
Was this method trained on Firsthand data?
No. Firsthand is not affiliated with the authors, and no named model was trained on Firsthand data. Robot demonstrations, teleoperation data and sub-goal images are not Firsthand deliverables.
How can I request a fast, on-demand custom egocentric dataset for this research?
Request a custom dataset with your tasks and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep.