Glossary
What Is "Giving Robots a Hand" (Eye-in-Hand Human Video Demonstrations)?
Last reviewed: November 2026
Short answer
Giving Robots a Hand is the title of arXiv 2307.05959, a framework that trains robot manipulation policies on eye-in-hand human video demonstrations. A fixed image mask hides the hand or gripper, an inverse dynamics model labels the human video with actions, and behavioral cloning learns from it. Third-party research; Firsthand is not affiliated.
01
What does the Giving Robots a Hand paper propose?
The paper, Giving Robots a Hand: Learning Generalizable Manipulation with Eye-in-Hand Human Video Demonstrations (arXiv 2307.05959), is by Moo Jin Kim, Jiajun Wu and Chelsea Finn of Stanford University. The authors note that collecting expert demonstrations with a real robot is expensive, while videos of humans performing tasks are much cheaper to collect and can be captured quickly in a wide range of scenarios. They augment narrow robot imitation datasets with broad unlabeled human video to improve the generalization of eye-in-hand visuomotor policies. The paper gives its approach no acronym, so this entry uses its title. See also the site’s wrist camera and human demonstration data entries.
02
Where does Firsthand fit next to this method?
This is third-party research. Firsthand is not affiliated with the authors, and no named model, including this one, was trained on Firsthand data. Firsthand does not claim to use a Franka Emika Panda arm, an Oculus Quest controller, CycleGAN, PyTorch or any hardware or software the paper names. The site states custom egocentric collection, fast and on demand: first-person human video with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, with one task instruction per episode. Per the site’s wrist camera entry, Firsthand captures a head-mounted global-shutter camera plus two wrist cameras. The paper’s camera is a single RGB camera strapped to the demonstrator’s forearm, and the site does not say Firsthand uses that setup. Image masking, inferred action labels, robot demonstrations, robot play data, teleoperation data, paired human-robot video and trained policies are not Firsthand deliverables.
03
How does the framework work?
Per the paper, there are three modules. First, an RGB camera is secured to the demonstrator’s forearm with two rubber bands, and the same camera is mounted on the robot arm. Second, the authors capture 100 by 100 images and zero out the top 36 rows of pixels in both domains, so the hand or end-effector is no longer visible. They argue this closes the visual gap without explicit domain adaptation. Third, an inverse dynamics model trained on robot play data predicts actions from pairs of consecutive masked images, and those inferred actions label the human video. A behavioral cloning policy then trains on robot and human demonstrations, conditioned on a binary grasp state that stands in for the proprioceptive information the mask removes. The second table lists the components.
04
What did the authors test, and what data did they use?
The authors evaluate eight real-world tasks with 3-DoF and 6-DoF arm control on a Franka Emika Panda arm, teleoperated with an Oculus Quest virtual reality controller for robot data. Environment generalization covers reaching, cube grasping, plate clearing and toy packing. Task generalization covers cube stacking, cube pick-and-place, plate clearing and toy packing. Four methods are compared: robot demonstrations only, robot plus play data, robot plus CycleGAN-translated human demonstrations, and robot plus masked human demonstrations.
The abstract reports a 58% absolute average improvement in success rate. In the authors’ Table 1, the masked method averages 63.64% on environment generalization and 58.75% on task generalization, against 5.23% and 0% for robot demonstrations only. These are the authors’ figures; Firsthand has not verified them. The datasets are the authors’ own. The sources read name no public dataset, and any dataset or code licence is Not stated in the sources read.
05
What limits do the authors state?
Per the paper, masking places some limits on the tasks that can be performed. The method may be less effective when a target object is so small (under 1.5 cm on each side) that it is not visible in the unmasked part of the image. It also needs a robot play dataset to train the inverse model, which the authors call inexpensive and hope to automate. They add that one could skip play data by inferring actions with visual odometry or Structure-from-Motion pose estimation. The authors also say the paper is not a head-to-head comparison of eye-in-hand and third-person methods.
06
How does it differ from related entries on this site?
The third table sets this method beside six site entries. It uses only what those entries state and what the sources read state, and it makes no performance comparison.
07
What would this kind of approach need from custom collection?
Per the paper, the approach needs broad human video recorded with an eye-in-hand camera, plus narrow robot demonstrations, robot play data and a robot, all the buyer’s own. Firsthand covers a different input: first-person human video, as custom collection at scale through a network of thousands of capture providers, for a consistent, recurring data flow to labs, with a first validated batch in a median of 48 hours where coverage is deep. Whether that video suits an eye-in-hand masking setup is the buyer’s own research question.
08
Giving Robots a Hand, as stated in the sources read
| Item | What the sources state |
|---|---|
| Paper | Giving Robots a Hand: Learning Generalizable Manipulation with Eye-in-Hand Human Video Demonstrations (arXiv 2307.05959) |
| Authors | Moo Jin Kim, Jiajun Wu, Chelsea Finn; Stanford University |
| Method name | No acronym; the paper’s title is used here |
| Human camera setup | An RGB camera secured to the demonstrator’s forearm with two rubber bands |
| Domain gap | Closed by masking a fixed region of every image, with no explicit domain adaptation method |
| Human action labels | Inferred by an inverse dynamics model trained on robot play data |
| Policy | Behavioral cloning on robot and human demonstrations, conditioned on grasp state |
| Robot | Franka Emika Panda arm; robot data teleoperated with an Oculus Quest controller |
| Official project page | giving-robots-a-hand.github.io, listed in the sources below |
| Licences | Not stated in the sources read |
| Firsthand relationship | None |
10
Pipeline components and ablations, per the paper
| Component | What the paper states |
|---|---|
| Eye-in-hand camera | Mounted on the human forearm and on the robot arm, so both domains view the scene from the hand |
| Image mask | 100 by 100 images; the top 36 rows are zeroed by a script in both domains |
| Robot play data | Task-agnostic robot data, collected in a similar manner as prior play-data work, used to train the inverse model |
| Inverse dynamics model | A convolutional network; takes two consecutive masked images and predicts the action between them |
| Behavioral cloning policy | Takes one masked image and the grasp state; trained on robot and human demonstrations together |
| Grasp state | A binary open or closed variable, set from the previous timestep’s inferred grasp action |
| CycleGAN baseline | Translates human images into robot images; the authors report it was noisier than masking |
| Ablations (authors’ Table 2) | Masked method 54.29%; no mask 24.29%; no grasp state 28.57%, as the authors report |
11
This method beside related entries on the site
| Site entry | What the entry states | How this method differs (sources read) |
|---|---|---|
| Wrist camera | A camera at the wrist that keeps the grasp point in frame | The paper mounts one camera on the human forearm and the robot arm, then masks part of the image |
| EgoMimic | Co-trains robot policies on egocentric human video with 3D hand tracking and on robot data | Here human video has no hand tracking; actions are inferred by an inverse dynamics model from masked images |
| DexCap | A portable hand-motion capture system and imitation method for dexterous robots | Here the human input is video from a forearm camera, and the robot end-effector is a parallel-jaw gripper |
| Universal Manipulation Interface | A handheld-gripper framework for collecting robot demonstrations without a robot | The paper contrasts its approach with reacher-grabber tools and records demonstrations with the bare human hand |
| AVID | Translates human demonstration images into robot images with CycleGAN, then learns each stage with model-based reinforcement learning | The paper compares against CycleGAN translation and reports masking performed better |
| Robotic Telekinesis | Teleoperates a robot hand and arm from one RGB camera, using a retargeter trained on human hand video | The paper cites it among hand-pose retargeting works; it avoids that route with a parallel-jaw gripper |
12
Frequently asked questions
- What is Giving Robots a Hand?
- It is the title of arXiv 2307.05959, a framework that adds broad eye-in-hand human video demonstrations to narrow robot demonstrations to train behavioral cloning policies. It is third-party research.
- What does eye-in-hand mean here?
- The camera moves with the hand. In the paper it is secured to the human forearm and mounted on the robot arm, so human and robot observations share the same viewpoint style.
- Why does the method mask part of every image?
- Masking a fixed region hides the human hand or robot end-effector, which the authors say closes the visual gap between human and robot data without an explicit domain adaptation method.
- Where do the human actions come from?
- Human video has no action labels, so the authors infer them with an inverse dynamics model trained on robot play data. Firsthand does not deliver inferred action labels.
- Does the method use third-person video?
- The policy input is the masked eye-in-hand image only. Firsthand captures first-person human video and does not capture third-person or fixed external-camera footage.
- Was this method trained on Firsthand data?
- No. Firsthand is not affiliated with the authors, and no named model was trained on Firsthand data. Robot demonstrations, play data, teleoperation data and paired human-robot video are not Firsthand deliverables.
- How can I request a fast, on-demand custom egocentric dataset for this research?
- Request a custom dataset with your tasks and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep.
Related terms
Sources
- Giving Robots a Hand: Learning Generalizable Manipulation with Eye-in-Hand Human Video Demonstrations (arXiv 2307.05959)
- Official project page: Giving Robots a Hand
- AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos (arXiv 1912.04443)
- Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on YouTube (arXiv 2202.10448)