Now recruiting capture teams worldwide. Get paid to record first-person video of real work.

Apply to capture

Glossary

What Is Imitation from Observation (Context Translation)?

A robot learns from observation-only video by translating the demonstration into its own context. Third-party; Firsthand is not affiliated.

Last reviewed: November 2026

Short answer

Imitation from observation is the problem and method named in arXiv 1707.03374: a robot learns a policy only from video observations of a behavior, with no demonstrated actions, even when the demonstration differs in viewpoint, objects or environment. The authors translate the demonstration into the robot’s context and track it with reinforcement learning. Third-party research; Firsthand is not affiliated.

01

What is imitation from observation, and what problem does it address?

The paper, Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translation (arXiv 1707.03374), is by YuXuan Liu, Abhishek Gupta, Pieter Abbeel and Sergey Levine of UC Berkeley. Standard imitation learning assumes the agent receives observation-action tuples. The authors contrast this with how people imitate: we watch someone perform a behavior and then work out which actions will realize it, compensating for changes in viewpoint, surroundings and object positions. They call this imitation-from-observation: learning policies only from a sequence of observations of the desired behavior, each sequence recorded under a different context, meaning changes in the environment, the objects or the viewpoint. See also the site’s imitation learning entry.

02

Where does Firsthand fit next to this method?

This is third-party research. Firsthand is not affiliated with the authors, and no named model, including this one, was trained on Firsthand data. Firsthand does not claim to use a Sawyer robot, the MuJoCo simulator or any software the paper names. The site states custom egocentric collection, fast and on demand: first-person human video with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, with one task instruction per episode. Per the site’s egocentric vs third-person comparison, Firsthand does not capture third-person or fixed external-camera footage, although the paper’s example translates from a third-person human demonstration. Translated demonstrations, robot demonstrations, teleoperation data, paired human-robot video, reward functions, trained models and policies are not Firsthand deliverables.

03

How does the context translation model work?

Per the paper, the model is trained on pairs of demonstrations, one from a source context and one from a target context. It must output the target-context observations, given the source demonstration and the first observation of the target context. The authors assume the paired demonstrations are aligned in time and say this could be relaxed in future work. The model has four components: two encoders, a translator and a decoder, with skip connections from the second encoder to the decoder. Once trained, it can translate any demonstration into the learner’s own context. The second table lists the components and losses.

04

How does a translated demonstration become a policy?

The reward penalizes the squared Euclidean distance between the encoding of the learner’s current observation and the average of the translated demonstration features. A weak image-tracking reward on the translated observations is added, weighted by a small constant. The authors say any reinforcement learning algorithm can be used. They use trust region policy optimization (TRPO) in simulation, and for the real robot the trajectory-centric method from guided policy search (GPS), because TRPO has high sample complexity.

05

What did the authors test, and what data did they use?

Simulated experiments in the MuJoCo simulator covered reaching, pushing, sweeping and striking, with demonstrations from a policy trained on a hand-specified reward. Real-world experiments used a 7-DoF Sawyer robot to push a cylinder, sweep crumpled paper or almonds, and ladle almonds, with every demonstration provided by a human. Compared methods were reward learning on pre-trained Inception-v3 features, third person imitation learning and generative adversarial imitation learning. The authors report that these struggled, and that on ladling almonds their method achieved a success rate of 66%. These are the authors’ figures; Firsthand has not verified them.

The project page lists the number of training videos for each task. The paper names no public human-video dataset, and any dataset or code licence is Not stated in the sources read.

06

What limits do the authors state?

Per the paper, the method needs a substantial number of demonstrations to learn the translation model, needs demonstrations from multiple contexts, and trains an end-to-end model from scratch for each task. It assumes the learner’s context comes from the same distribution as the demonstrations. In the real-world tasks the human and the robot used the same tool, and videos were cropped around task-relevant areas, which avoided systematic differences between a human arm and a robot arm. Large embodiment differences are left to future work.

07

How does it differ from related entries on this site?

The third table sets this method beside five site entries. It uses only what those entries state and what the sources read state, and it makes no performance comparison.

08

What would this kind of approach need from custom collection?

Per the paper, the approach needs video demonstrations of a task from several contexts, plus a robot and a reinforcement learning stack, all the buyer’s own. Firsthand covers a different input: first-person human video, as custom collection at scale through a network of thousands of capture providers, for a consistent, recurring data flow to labs, confirmed on a Spec Pilot. Whether first-person video suits a given context translation setup is the buyer’s own research question.

09

Imitation from observation via context translation, as stated in the sources read

Imitation from observation via context translation, as stated in the sources read
ItemWhat the sources state
PaperImitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translation (arXiv 1707.03374)
AuthorsYuXuan Liu, Abhishek Gupta, Pieter Abbeel, Sergey Levine; UC Berkeley
Problem namedImitation-from-observation: learn policies only from observations of the desired behavior, recorded under differences in context
MethodVideo prediction with context translation, plus deep reinforcement learning
Demonstration actionsNot observed; the demonstrations contain observations only
RewardSquared distance to translated demonstration features, plus a weak image-tracking term
Reinforcement learningTRPO in simulation; trajectory-centric guided policy search (GPS) on the real robot
Simulation tasksReaching, pushing, sweeping and striking, in the MuJoCo simulator
Real robot and tasksA 7-DoF Sawyer; pushing, sweeping paper or almonds, and ladling almonds
Official project pageLinked from the paper and listed in the sources below
LicencesNot stated in the sources read
Firsthand relationshipNone

11

Context translation model: components and losses, per the paper

Context translation model: components and losses, per the paper
Component or lossWhat the paper states
Source observation encoder (Enc1)Encodes an observation from the source demonstration into features z1; also encodes the learner’s observations for the reward
Target initial observation encoder (Enc2)Encodes the first observation of the target context into features z2
Translator TProduces z3 = T(z1, z2), translating the features into the context of z2
Decoder (Dec)Decodes z3 into the predicted target-context observation, with skip connections from Enc2
Translation lossSquared error between the predicted and true target-context observation
Reconstruction lossTrains Enc1 and Dec jointly as an autoencoder
Alignment lossAligns encoded target observations with the translated features z3, so tracked features share one space
WeightsTwo hyperparameters, λ1 and λ2, weight the reconstruction and alignment losses; the encoders may have tied or different weights

12

This method beside related entries on the site

This method beside related entries on the site
Site entryWhat the entry statesHow this method differs (sources read)
Imitation learningTraining a policy to reproduce demonstrated behavior from example trajectoriesThe demonstrations here contain observations only, with no actions, so direct behavioral cloning cannot be used
AVIDTranslates human demonstration images into robot images with CycleGAN, then learns each stage with model-based reinforcement learningHere an encoder, translator and decoder model translates between contexts, and the paper says it does not use GANs
Third-person visual imitationOne third-person human video; a high-level module sets first-person sub-goals and a low-level controller actsHere translated features define a reward for reinforcement learning; the sources read describe no separate sub-goal controller
DAMLMeta-learns a prior, then learns a task from one human videoThe sources read describe a translation model trained on demonstrations from many contexts, with no meta-learning
Time-Contrastive NetworksLearns a viewpoint-invariant embedding from unlabeled multi-view videoThe paper cites multi-viewpoint training of visual features as work that raises invariance; this method adds an explicit context translation model

13

Frequently asked questions

What is imitation from observation?
Per arXiv 1707.03374, it is learning policies only from observations of a desired behavior, with no demonstrated actions, under differences in context such as viewpoint, objects and environment. It is third-party research.
What is context translation in this paper?
A model that converts a demonstration from one context to another, conditioned on the first observation in the target context. The features it learns are then used to define a reward for reinforcement learning.
Does the method need demonstrated actions?
No. The paper states the actions are not observed in the demonstrations, so direct behavioral cloning cannot be used.
Which robot and tasks does the paper report?
A 7-DoF Sawyer for pushing, sweeping paper or almonds, and ladling almonds, plus four simulated tasks in MuJoCo: reaching, pushing, sweeping and striking.
Does the method use first-person video?
The paper’s example translates from a third-person human demonstration into a first-person robot context. Firsthand captures first-person human video and does not capture third-person footage.
Was this method trained on Firsthand data?
No. Firsthand is not affiliated with the authors, and no named model was trained on Firsthand data. Translated demonstrations, robot demonstrations and paired human-robot video are not Firsthand deliverables.
How can I request a fast, on-demand custom egocentric dataset for this research?
Request a custom dataset with your tasks and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep.