Glossary
What Is Robotic Telekinesis (Learning a Robot Hand Imitator by Watching Humans on YouTube)?
Last reviewed: November 2026
Short answer
Robotic Telekinesis is the system in arXiv 2202.10448. A single uncalibrated RGB camera watches a person move their hand and arm, and the robot imitates in real time. The authors train a hand retargeter on human hand images from internet video, with no paired human-robot data. Third-party research; Firsthand is not affiliated.
01
What is Robotic Telekinesis, and what problem does it address?
Robotic Telekinesis is named in Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtube (arXiv 2202.10448) by Aravind Sivakumar, Kenneth Shaw and Deepak Pathak of Carnegie Mellon University, published at Robotics: Science and Systems 2022. The authors study teleoperation of a dexterous robot hand. They say existing options such as virtual reality devices, haptic gloves and motion capture involve expensive hardware, expert operators or an apparatus that impedes natural hand motion, and that DexPilot relies on multiple calibrated depth cameras. Their goal is a low-cost, glove-free, marker-free system that works for untrained operators with one uncalibrated camera. The central obstacle is that paired human-robot hand data is expensive to collect, while human hand video is abundant.
02
Where does Firsthand fit next to Robotic Telekinesis?
Robotic Telekinesis is third-party research. Firsthand is not affiliated with the authors, and no named model, Robotic Telekinesis included, was trained on Firsthand data. Firsthand does not claim to use OpenPose, FrankMocap, MANO, SMPL-X, an xArm6, an Allegro hand or any tool the paper names. The site states custom egocentric collection, fast and on demand: first-person human video with calibrated stereo depth, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, with one task instruction per episode. Teleoperation, retargeting or robot-hand control data, robot joint angles and trained retargeters are not Firsthand deliverables.
03
How does Robotic Telekinesis turn a human hand into robot hand commands?
Per the paper, each camera image is split into a hand branch and an arm branch. For the hand, an OpenPose-derived detector crops the operator’s hand and a FrankMocap pose estimator returns parameters of a 3D MANO hand model. A small neural network, the hand retargeter, maps those parameters to the 16 joint angles of an Allegro hand.
Because no ground-truth robot poses exist for human hand images, the authors train the retargeter with an energy function instead of labels. It compares five keypoints (four fingertips and the palm) and the ten vectors between them on the human and robot hands. A frozen self-collision classifier is added as an adversarial loss, so predicted poses avoid fingers colliding with each other or the palm.
04
How is the robot arm controlled from the same camera?
For the arm, a body pose estimator from FrankMocap models the operator with SMPL-X. Without a depth sensor or camera calibration, the authors use the pose of the human right wrist relative to the torso as the target for the robot wrist relative to a chosen robot torso point. They reject outliers, apply a low-pass filter to the wrist poses and use an inverse kinematics solver for the arm joints.
05
What YouTube and human video does Robotic Telekinesis learn from?
The authors describe a dataset of about 20 million images from Epic Kitchens and 100 Days of Hands, a collection of YouTube videos, with hand poses estimated by the same estimator used at deployment. They add FreiHand, a smaller dataset with ground-truth hand poses. The first table lists what the paper states. The licence of these datasets is Not stated in the sources read; check each official source before use.
06
What did the authors report, and what are the limits of this page?
The second table repeats figures the authors report; Firsthand has not verified them. The authors state limits: a single colour camera causes failures tied to occlusion and ambiguous depth, the thumb of the Allegro hand was retargeted less accurately, and the hand tracked worse when fully open with the palm parallel to the camera. They also call the system a starting point for research rather than an end product.
07
How does Robotic Telekinesis differ from related methods on this site?
The third table sets Robotic Telekinesis beside five site entries. The comparison uses only what those entries state and what the paper states.
08
What would a Robotic Telekinesis-style approach need from custom collection?
Per the paper, the method needs diverse human hand images for the retargeter and a robot arm and hand, both the buyer’s own. The paper draws on existing public video and does not describe collecting new footage. Firsthand covers the human video side only, as custom collection at scale through a network of thousands of capture providers, for a consistent, recurring data flow to labs, confirmed on a Spec Pilot. The 21-joint 3D hand pose the site states is a different representation from the MANO parameters the paper uses.
09
Robotic Telekinesis at a glance, as stated in the paper and project page
| Item | What the sources state |
|---|---|
| Paper | Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtube (arXiv 2202.10448) |
| Method name | Robotic Telekinesis |
| Authors and venue | Aravind Sivakumar, Kenneth Shaw, Deepak Pathak; Carnegie Mellon University; Robotics: Science and Systems 2022 |
| Input | A single uncalibrated RGB camera viewing the operator; glove-free and marker-free |
| Hand pipeline | OpenPose-derived crop, FrankMocap hand pose (MANO), neural hand retargeter to 16 Allegro joint angles |
| Arm pipeline | FrankMocap body pose (SMPL-X), wrist-to-torso transform, outlier rejection, low-pass filter, inverse kinematics |
| Robot | An xArm6 arm with a 16-DoF Allegro hand |
| Licences | Not stated in the sources read |
| Firsthand relationship | None |
11
Human video and hand data stated by the authors (as written in the paper)
| Source | What the paper states |
|---|---|
| Epic Kitchens | Egocentric videos of humans performing daily household tasks; part of about 20 million images gathered |
| 100 Days of Hands | A collection of YouTube videos, links only; hand poses estimated by the paper’s own estimator |
| FreiHand | A small, clean dataset with ground-truth hand poses, used to augment the noisy estimated poses |
| Retargeter training set | The appendix lists 30,000 samples from FreiHand and 30,000 from 100 Days of Hands |
| Paired human-robot data | None used; the authors say it is expensive to collect |
12
Figures reported by the authors (Firsthand has not verified them)
| Measure | What the authors report |
|---|---|
| Trained operator study | Ten tasks, ten trials each, one-minute timeout; outperformed the DexPilot-Monocular baseline on 7 of 10 tasks and performed similarly on the other 3 |
| Novice operator study | Ten previously untrained operators, three tasks, seven trials each, about 15 minutes in total per person |
| Retargeting accuracy | RMSE against an oracle of 0.17 radians for the neural retargeter versus 0.25 radians for the baseline |
| Hand retargeter rate | About 24 Hz for the neural network versus 10 Hz for the online-optimisation baseline |
13
Robotic Telekinesis beside related entries on the site
| Site entry | What the entry states | How Robotic Telekinesis differs (sources read) |
|---|---|---|
| VideoDex | Pretrains robot-hand policies on retargeted human video | Robotic Telekinesis is a real-time teleoperation system; it retargets each camera frame to commands and does not train a manipulation policy |
| DexMV | Turns human hand-object video into multi-finger robot-hand demonstrations for imitation learning | Robotic Telekinesis targets live control of a robot by an operator, not demonstration generation from recorded video |
| Motion retargeting | Maps captured human hand and wrist motion onto a robot with different kinematics | Robotic Telekinesis is one such mapping, learned from human video with an energy-based loss |
| Hand pose estimation | Recovers 3D positions of hand and finger joints across a clip | The paper uses an existing estimator as its first stage and does not propose a new one |
| Teleoperation vs egocentric data | Compares robot-collected demonstrations with human-worn first-person capture | Robotic Telekinesis is a teleoperation method that removes gloves and markers, using a camera instead |
14
Frequently asked questions
- What is Robotic Telekinesis?
- Per arXiv 2202.10448, it is a system for real-time visual teleoperation of a robot hand and arm from one RGB camera, with a hand retargeter trained on human hand images from internet video. It is third-party research.
- Does Robotic Telekinesis need gloves, markers or camera calibration?
- No. The authors describe the system as low-cost, glove-free and marker-free, and it uses a single uncalibrated colour camera.
- How does it learn without paired human-robot data?
- The retargeter is trained to minimise an energy function that compares fingertip and palm keyvectors on the human and robot hands, plus a self-collision classifier used as an adversarial loss. No robot labels are needed.
- What human video did the authors use, and is it licensed?
- Hand images from Epic Kitchens and 100 Days of Hands, plus FreiHand for ground-truth poses. The licence is Not stated in the sources read; check each dataset’s official page.
- Is this the same as vision-based dexterous hand retargeting?
- It is one example. The paper estimates a 3D hand pose from RGB and maps it to robot joint angles. The motion retargeting entry on this site describes the broader idea.
- Was Robotic Telekinesis trained on Firsthand data?
- No. Firsthand is not affiliated with the authors, and no named model was trained on Firsthand data. Retargeters, robot joint angles and teleoperation data are not Firsthand deliverables.
- How can I request a fast, on-demand custom egocentric dataset for this research?
- Request a custom dataset with your tasks and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep.
Related terms
Sources
- Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtube (arXiv 2202.10448)
- Robotic Telekinesis project page (official, linked from the paper)
- Scaling Egocentric Vision: The EPIC-KITCHENS Dataset (arXiv 1804.02748)
- EPIC-KITCHENS official site
- Understanding Human Hands in Contact at Internet Scale, the 100 Days of Hands paper (arXiv 2006.06669)
- FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Images (arXiv 1909.04349)
- FrankMocap official code (facebookresearch/frankmocap)
- DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm System (arXiv 1910.03135)