Now recruiting capture teams worldwide. Get paid to record first-person video of real work.

Apply to capture

Glossary

What Is Robotic Telekinesis (Learning a Robot Hand Imitator by Watching Humans on YouTube)?

Robotic Telekinesis lets a person teleoperate a robot hand and arm from one RGB camera, using a retargeter trained on human hand video. Third-party.

Last reviewed: November 2026

Short answer

Robotic Telekinesis is the system in arXiv 2202.10448. A single uncalibrated RGB camera watches a person move their hand and arm, and the robot imitates in real time. The authors train a hand retargeter on human hand images from internet video, with no paired human-robot data. Third-party research; Firsthand is not affiliated.

01

What is Robotic Telekinesis, and what problem does it address?

Robotic Telekinesis is named in Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtube (arXiv 2202.10448) by Aravind Sivakumar, Kenneth Shaw and Deepak Pathak of Carnegie Mellon University, published at Robotics: Science and Systems 2022. The authors study teleoperation of a dexterous robot hand. They say existing options such as virtual reality devices, haptic gloves and motion capture involve expensive hardware, expert operators or an apparatus that impedes natural hand motion, and that DexPilot relies on multiple calibrated depth cameras. Their goal is a low-cost, glove-free, marker-free system that works for untrained operators with one uncalibrated camera. The central obstacle is that paired human-robot hand data is expensive to collect, while human hand video is abundant.

02

Where does Firsthand fit next to Robotic Telekinesis?

Robotic Telekinesis is third-party research. Firsthand is not affiliated with the authors, and no named model, Robotic Telekinesis included, was trained on Firsthand data. Firsthand does not claim to use OpenPose, FrankMocap, MANO, SMPL-X, an xArm6, an Allegro hand or any tool the paper names. The site states custom egocentric collection, fast and on demand: first-person human video with calibrated stereo depth, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, with one task instruction per episode. Teleoperation, retargeting or robot-hand control data, robot joint angles and trained retargeters are not Firsthand deliverables.

03

How does Robotic Telekinesis turn a human hand into robot hand commands?

Per the paper, each camera image is split into a hand branch and an arm branch. For the hand, an OpenPose-derived detector crops the operator’s hand and a FrankMocap pose estimator returns parameters of a 3D MANO hand model. A small neural network, the hand retargeter, maps those parameters to the 16 joint angles of an Allegro hand.

Because no ground-truth robot poses exist for human hand images, the authors train the retargeter with an energy function instead of labels. It compares five keypoints (four fingertips and the palm) and the ten vectors between them on the human and robot hands. A frozen self-collision classifier is added as an adversarial loss, so predicted poses avoid fingers colliding with each other or the palm.

04

How is the robot arm controlled from the same camera?

For the arm, a body pose estimator from FrankMocap models the operator with SMPL-X. Without a depth sensor or camera calibration, the authors use the pose of the human right wrist relative to the torso as the target for the robot wrist relative to a chosen robot torso point. They reject outliers, apply a low-pass filter to the wrist poses and use an inverse kinematics solver for the arm joints.

05

What YouTube and human video does Robotic Telekinesis learn from?

The authors describe a dataset of about 20 million images from Epic Kitchens and 100 Days of Hands, a collection of YouTube videos, with hand poses estimated by the same estimator used at deployment. They add FreiHand, a smaller dataset with ground-truth hand poses. The first table lists what the paper states. The licence of these datasets is Not stated in the sources read; check each official source before use.

06

What did the authors report, and what are the limits of this page?

The second table repeats figures the authors report; Firsthand has not verified them. The authors state limits: a single colour camera causes failures tied to occlusion and ambiguous depth, the thumb of the Allegro hand was retargeted less accurately, and the hand tracked worse when fully open with the palm parallel to the camera. They also call the system a starting point for research rather than an end product.

07

How does Robotic Telekinesis differ from related methods on this site?

The third table sets Robotic Telekinesis beside five site entries. The comparison uses only what those entries state and what the paper states.

08

What would a Robotic Telekinesis-style approach need from custom collection?

Per the paper, the method needs diverse human hand images for the retargeter and a robot arm and hand, both the buyer’s own. The paper draws on existing public video and does not describe collecting new footage. Firsthand covers the human video side only, as custom collection at scale through a network of thousands of capture providers, for a consistent, recurring data flow to labs, confirmed on a Spec Pilot. The 21-joint 3D hand pose the site states is a different representation from the MANO parameters the paper uses.

09

Robotic Telekinesis at a glance, as stated in the paper and project page

Robotic Telekinesis at a glance, as stated in the paper and project page
ItemWhat the sources state
PaperRobotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtube (arXiv 2202.10448)
Method nameRobotic Telekinesis
Authors and venueAravind Sivakumar, Kenneth Shaw, Deepak Pathak; Carnegie Mellon University; Robotics: Science and Systems 2022
InputA single uncalibrated RGB camera viewing the operator; glove-free and marker-free
Hand pipelineOpenPose-derived crop, FrankMocap hand pose (MANO), neural hand retargeter to 16 Allegro joint angles
Arm pipelineFrankMocap body pose (SMPL-X), wrist-to-torso transform, outlier rejection, low-pass filter, inverse kinematics
RobotAn xArm6 arm with a 16-DoF Allegro hand
LicencesNot stated in the sources read
Firsthand relationshipNone

11

Human video and hand data stated by the authors (as written in the paper)

Human video and hand data stated by the authors (as written in the paper)
SourceWhat the paper states
Epic KitchensEgocentric videos of humans performing daily household tasks; part of about 20 million images gathered
100 Days of HandsA collection of YouTube videos, links only; hand poses estimated by the paper’s own estimator
FreiHandA small, clean dataset with ground-truth hand poses, used to augment the noisy estimated poses
Retargeter training setThe appendix lists 30,000 samples from FreiHand and 30,000 from 100 Days of Hands
Paired human-robot dataNone used; the authors say it is expensive to collect

12

Figures reported by the authors (Firsthand has not verified them)

Figures reported by the authors (Firsthand has not verified them)
MeasureWhat the authors report
Trained operator studyTen tasks, ten trials each, one-minute timeout; outperformed the DexPilot-Monocular baseline on 7 of 10 tasks and performed similarly on the other 3
Novice operator studyTen previously untrained operators, three tasks, seven trials each, about 15 minutes in total per person
Retargeting accuracyRMSE against an oracle of 0.17 radians for the neural retargeter versus 0.25 radians for the baseline
Hand retargeter rateAbout 24 Hz for the neural network versus 10 Hz for the online-optimisation baseline

13

Robotic Telekinesis beside related entries on the site

Robotic Telekinesis beside related entries on the site
Site entryWhat the entry statesHow Robotic Telekinesis differs (sources read)
VideoDexPretrains robot-hand policies on retargeted human videoRobotic Telekinesis is a real-time teleoperation system; it retargets each camera frame to commands and does not train a manipulation policy
DexMVTurns human hand-object video into multi-finger robot-hand demonstrations for imitation learningRobotic Telekinesis targets live control of a robot by an operator, not demonstration generation from recorded video
Motion retargetingMaps captured human hand and wrist motion onto a robot with different kinematicsRobotic Telekinesis is one such mapping, learned from human video with an energy-based loss
Hand pose estimationRecovers 3D positions of hand and finger joints across a clipThe paper uses an existing estimator as its first stage and does not propose a new one
Teleoperation vs egocentric dataCompares robot-collected demonstrations with human-worn first-person captureRobotic Telekinesis is a teleoperation method that removes gloves and markers, using a camera instead

14

Frequently asked questions

What is Robotic Telekinesis?
Per arXiv 2202.10448, it is a system for real-time visual teleoperation of a robot hand and arm from one RGB camera, with a hand retargeter trained on human hand images from internet video. It is third-party research.
Does Robotic Telekinesis need gloves, markers or camera calibration?
No. The authors describe the system as low-cost, glove-free and marker-free, and it uses a single uncalibrated colour camera.
How does it learn without paired human-robot data?
The retargeter is trained to minimise an energy function that compares fingertip and palm keyvectors on the human and robot hands, plus a self-collision classifier used as an adversarial loss. No robot labels are needed.
What human video did the authors use, and is it licensed?
Hand images from Epic Kitchens and 100 Days of Hands, plus FreiHand for ground-truth poses. The licence is Not stated in the sources read; check each dataset’s official page.
Is this the same as vision-based dexterous hand retargeting?
It is one example. The paper estimates a 3D hand pose from RGB and maps it to robot joint angles. The motion retargeting entry on this site describes the broader idea.
Was Robotic Telekinesis trained on Firsthand data?
No. Firsthand is not affiliated with the authors, and no named model was trained on Firsthand data. Retargeters, robot joint angles and teleoperation data are not Firsthand deliverables.
How can I request a fast, on-demand custom egocentric dataset for this research?
Request a custom dataset with your tasks and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep.