Glossary
What Is SFV (Reinforcement Learning of Physical Skills from Videos)?
Last reviewed: November 2026
Short answer
SFV, short for skills from videos, is the method in arXiv 1810.03599, “SFV: Reinforcement Learning of Physical Skills from Videos”. It lets physically simulated characters learn skills from ordinary monocular video by combining deep pose estimation, motion reconstruction and deep reinforcement learning imitation. Third-party research from UC Berkeley; Firsthand is not affiliated.
01
What does the SFV paper propose?
The paper, SFV: Reinforcement Learning of Physical Skills from Videos (arXiv 1810.03599), is by Xue Bin Peng, Angjoo Kanazawa, Jitendra Malik, Pieter Abbeel and Sergey Levine of the University of California, Berkeley, and appeared in ACM Transactions on Graphics in November 2018. The authors note that motion capture remains the most popular source of motion data but typically requires heavily instrumented environments and actors, while publicly available web video, such as clips from YouTube, is abundant. Their goal is to let physically simulated characters learn skills from that video, so a character controller can be designed simply by querying for recordings of the desired behavior. The paper names the method SFV, for skills from videos. See also the site’s motion retargeting and body pose estimation entries.
02
Where does Firsthand fit next to SFV?
SFV is third-party research. Firsthand is not affiliated with the authors, and no named model, including SFV, was trained on Firsthand data. Firsthand does not claim to use OpenPose, Human Mesh Recovery, the Bullet physics engine, TensorFlow, a simulated Atlas robot or any software the paper names. SFV starts from ordinary video rather than motion capture. The site states a different input: custom egocentric collection, fast and on demand, from a head-mounted global-shutter camera plus two wrist cameras on one PTP clock, with 3D hand pose at 21 joints per hand, 3D body pose synchronized with hand pose, depth and RGB, and one task instruction per episode. Pose estimated from video, reconstructed reference motion, simulated characters, trained policies, retargeted data, robot demonstrations, teleoperation data and paired human-robot video are not Firsthand deliverables.
03
How does SFV work?
Per the paper, a video clip and a simulated character model go in, and a control policy comes out, through three stages. In pose estimation, a 2D estimator (OpenPose) and a 3D estimator (Human Mesh Recovery, or HMR) are applied to every frame. The authors augment the training images with rotated copies, with rotations sampled uniformly between 0 and 2π, because acrobatic poses are underrepresented in standard datasets. In motion reconstruction, a 3D pose trajectory is optimized in the 3D estimator’s latent pose space to agree with both the 2D and 3D predictions and to stay consistent between adjacent frames, which mitigates jitter and sudden changes in pose. In motion imitation, a feedforward policy is trained with a variant of proximal policy optimization (PPO) and a motion imitation objective. The second table lists the components.
The authors’ main reinforcement learning extension is adaptive state initialization (ASI). Rather than starting every episode from frames of the reference motion, a second agent learns the distribution of initial states, which the authors describe as a form of automatic curriculum generation. They compare it with fixed and reference state initialization and report that ASI consistently improves performance and learning speed.
04
What video and characters did the authors use?
All clips were collected from YouTube, per the paper. They show human actors performing acrobatic stunts such as flips and cartwheels, and locomotion skills such as walking and running. The authors chose clips where the camera is primarily stationary and a single actor is present, trimmed each to the relevant motion, and used one demonstration of each skill. Their Table 1 summarizes over 20 skills.
Policies control a simulated humanoid character and, separately, a simulated Atlas robot whose proportions differ significantly from human proportions. Physics runs in the Bullet engine, and the authors say each policy takes approximately 150 million samples, about one day on a 16-core machine. These are the authors’ figures; Firsthand has not verified them. The sources read describe no real robot. They also show motion completion, which predicts plausible future motion for an actor from a single still image by running learned controllers. The sources read name no dataset or code licence, so any licence is Not stated in the sources read.
05
What limits do the authors state?
Per the paper, success depends on the accuracy of the reconstructed motion. When the pose estimators fail to predict an actor’s pose, as in the kip-up and spinkick examples, the resulting policy fails to reproduce the behavior. Characters still sometimes show peculiar postures and stiff movements, and fast dance steps, such as the Gangnam Style clip, remain challenging. Difficulty estimating the global translation of the root limited results primarily to clips with minimal camera motion, and the authors say a large variety of video clips remain beyond what their system can imitate.
06
How does SFV differ from related entries on this site?
The third table sets SFV beside four site entries. It uses only what those entries state and what the sources read state, and it makes no performance comparison.
07
What would this kind of approach need from custom collection?
Per the paper, SFV needs monocular video of an actor, a simulated character model and a physics simulator, all the buyer’s own, plus a pose-estimation stage that Firsthand does not supply. Firsthand covers a different input: first-person human video with captured hand and body pose, as custom collection at scale through a network of thousands of capture providers, for a consistent, recurring data flow to labs, with a first validated batch in a median of 48 hours where coverage is deep. Whether that data suits a video-imitation pipeline is the buyer’s own research question.
08
SFV, as stated in the sources read
| Item | What the sources state |
|---|---|
| Paper | SFV: Reinforcement Learning of Physical Skills from Videos (arXiv 1810.03599) |
| Authors | Xue Bin Peng, Angjoo Kanazawa, Jitendra Malik, Pieter Abbeel, Sergey Levine; University of California, Berkeley |
| Venue | ACM Transactions on Graphics, Vol. 37, No. 6, Article 178 (November 2018); the project page lists SIGGRAPH Asia 2018 |
| Method name | SFV, for skills from videos |
| Input | A monocular video clip and a simulated character model |
| Pose estimators | OpenPose for 2D pose and Human Mesh Recovery (HMR) for 3D pose |
| Policy learning | A proximal policy optimization (PPO) variant with adaptive state initialization |
| Characters | A simulated humanoid and a simulated Atlas robot |
| Official project page | xbpeng.github.io/projects/SFV, listed in the sources below |
| Licences | Not stated in the sources read |
| Firsthand relationship | None |
10
Pipeline components, per the paper
| Component | What the paper states |
|---|---|
| Pose estimation | OpenPose gives 2D pose and HMR gives 3D pose, applied independently to every frame; training images are augmented with rotated copies |
| Motion reconstruction | A 3D pose trajectory is optimized in the 3D estimator’s latent pose space against a 2D consistency term, a 3D consistency term and a smoothness term |
| Motion imitation | A feedforward policy trained with a PPO variant on a motion imitation objective |
| Adaptive state initialization | A learned initial state distribution, compared by the authors with fixed and reference state initialization |
| Characters | A simulated humanoid with 12 joints and a simulated Atlas robot |
| Simulation | Bullet physics engine at 1.2 kHz with the policy queried at 30 Hz; networks built and trained in TensorFlow |
| Video | Clips collected from YouTube; camera primarily stationary, a single actor, one demonstration per skill |
| Applications | Retargeting skills to different characters and environments, and motion completion from a single still image |
11
SFV beside related entries on the site
| Site entry | What the entry states | How SFV differs (sources read) |
|---|---|---|
| Motion retargeting | Maps captured human hand and wrist motion onto a robot with different kinematics | SFV retargets by training a policy with reinforcement learning on reconstructed motion; its retargeting example is a simulated Atlas robot, not a robot hand |
| Body pose estimation | Recovering 3D positions of body joints over time, alongside hand pose | SFV estimates body pose from monocular video with OpenPose and HMR, then reconstructs the motion; Firsthand captures body pose synchronized with hand pose, depth and RGB |
| HumanPlus | A published system for humanoid shadowing and imitation from human data, shadowing body and hand motion with only an RGB camera | SFV trains simulated characters to imitate clips; the sources read describe no shadowing of a real humanoid |
| OKAMI | Generates a manipulation plan from a single RGB-D human video and derives a policy for a humanoid, using object-aware retargeting | SFV starts from monocular video, the sources read describe no depth input, and it imitates full-body skills in simulation |
12
Frequently asked questions
- What is SFV?
- SFV, short for skills from videos, is the method in arXiv 1810.03599. It trains physically simulated characters to imitate skills from monocular video using pose estimation, motion reconstruction and reinforcement learning. It is third-party research.
- How does SFV learn a skill from a video?
- It estimates the actor’s 2D and 3D pose in every frame, reconstructs a consistent reference motion, then trains a policy with reinforcement learning so a simulated character reproduces that motion in a physics simulation.
- Does SFV use a real robot?
- Not in the sources read. The policies control a simulated humanoid and a simulated Atlas robot in a physics simulation.
- Where does the reference motion come from?
- It is estimated from monocular video by OpenPose and HMR and then reconstructed by optimization. Firsthand does not deliver estimated or reconstructed reference motion; it captures hand and body pose.
- What limits do the authors state?
- Policies fail when the pose estimators miss the actor’s pose, fast dance steps remain challenging, and results are primarily limited to clips with minimal camera motion.
- Was SFV trained on Firsthand data?
- No. Firsthand is not affiliated with the authors, and no named model was trained on Firsthand data. Simulated characters, trained policies, retargeted data, robot demonstrations and teleoperation data are not Firsthand deliverables.
- How can I request a fast, on-demand custom egocentric dataset for this research?
- Request a custom dataset with your tasks and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep.
Related terms
Sources
- SFV: Reinforcement Learning of Physical Skills from Videos (arXiv 1810.03599)
- Official project page: SFV
- ACM Transactions on Graphics version of record (DOI 10.1145/3272127.3275014)
- End-to-end Recovery of Human Shape and Pose, the HMR paper (arXiv 1712.06584)
- Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields, the OpenPose paper (arXiv 1611.08050)