Glossary
What Is R+X (Retrieval and Execution from Everyday Human Videos)?
Last reviewed: November 2026
Short answer
R+X, Retrieval and Execution, is a framework from Imperial College London (arXiv 2407.12957) that teaches robots skills from long, unlabelled first-person videos of humans doing everyday tasks. Given a language command, it retrieves relevant clips with a vision language model, then conditions in-context imitation learning on them, with no training on those clips. Firsthand is not affiliated.
01
What is R+X, and what problem does it address?
R+X is named in R+X: Retrieval and Execution from Everyday Human Videos (arXiv 2407.12957) by Georgios Papagiannis, Norman Di Palo, Pietro Vitiello and Edward Johns of the Robot Learning Lab at Imperial College London, submitted on 17 July 2024 and published at ICRA 2025. The authors study robot skills from long, unlabelled, first-person videos of a person going about daily life. They remove assumptions of earlier work: videos aligned with robot videos, language labels, or demonstrations on robot or motion-capture hardware.
02
Where does Firsthand fit next to R+X?
Firsthand is not affiliated with the R+X authors. No named model, R+X included, was trained on Firsthand data, and Firsthand does not claim to use any hardware or software named in the sources: a chest-worn Intel RealSense 455, Gemini Pro 1.5 Flash, DINO, HaMeR, KAT, and a Sawyer robot with a Robotiq 2F-85 gripper. Firsthand captures real human egocentric video, custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license, with one task instruction per episode. Robot trajectories, retargeted actions, action labels, hand pose priors, 3D object models, simulation environments, co-training mixtures, pretrained representations, embeddings, synthetic data and language annotations beyond that instruction are not Firsthand deliverables.
03
How does R+X turn a long unlabelled video into a robot skill?
Per the paper, R+X takes three inputs: one long unlabelled human video, a language command and the robot’s current RGB-D observation. It outputs a trajectory of 6-DoF gripper poses. In retrieval, a vision language model, Gemini Pro 1.5 Flash, is prompted with the video and asked for the start and end seconds where the task happens, returning shorter clips.
In execution, each clip is reduced to sparse 3D data: visual keypoints from DINO descriptors and a 21-joint hand trajectory from HaMeR. These become examples for Keypoint Action Tokens (KAT), an in-context imitation method built on off-the-shelf language models, which predicts hand joints for the live scene. The authors state there is no training or finetuning, so skills run immediately.
04
What video does R+X learn from?
The authors recorded the human video with an Intel RealSense 455 worn on the chest, in many rooms and buildings, then used HaMeR to drop frames without visible hands. Because the camera moves, each clip is stabilised with CLIPSeg segmentation and TAPIR point tracking. The project page also reports chest, head and third-person views. The paper says R+X needs a single RGB-D camera and no extrinsics calibration.
The authors recorded their own video; no public human video dataset is named. Any video or code licence is Not stated in the sources read.
05
How do hand joints become gripper actions?
The authors apply heuristics by interaction type. For grasping they align the gripper fingertips with the index and thumb tips; for pressing and pushing they align points on a closed gripper with index finger joints. Gemini picks the heuristic from the command. They target a Robotiq 2F-85 on a Sawyer robot with a RealSense 415 head camera, and note a human-like dexterous hand would not need them. Firsthand does not claim to use any of this hardware.
06
What did the authors report, and what are the limits?
On 12 everyday tasks, with 10 evaluations per task, the authors report an average success of 0.7 for R+X against 0.55 for R3M-DiffLang and 0.53 for Octo, both fine-tuned baselines trained on Gemini-captioned clips. They also report strong performance with as few as 10 retrieved clips, and no drop on earlier tasks when new ones are added.
Stated limits: hand-pose errors limit precise tasks, keypoints can bottleneck with many similar objects, and there is a delay of a few seconds. These are the authors’ figures; Firsthand has not verified them.
07
How does R+X compare with related entries on this site?
The table sets R+X beside four site entries. R+X retrieves clips for each command and conditions in-context, rather than training a policy.
08
What would an R+X-style approach need from custom collection?
Per the sources read, it needs a long first-person video that includes hands, an RGB-D view, a vision language model and a robot, all the buyer’s own. The paper needs no labels or calibration, so a per-episode instruction is not a stated requirement.
This is custom collection: request on-demand collection and confirm it on a Spec Pilot, with a median of 48 hours to a first validated batch where coverage is deep and three to four weeks where it is thin.
09
R+X at a glance, as stated in the paper and project page
| Item | What the sources state |
|---|---|
| Paper | R+X: Retrieval and Execution from Everyday Human Videos (arXiv 2407.12957) |
| Authors | Georgios Papagiannis, Norman Di Palo, Pietro Vitiello, Edward Johns |
| Institution and venue | Imperial College London; ICRA 2025 |
| Retrieval | Gemini Pro 1.5 Flash returns start and end seconds of relevant clips |
| Execution | KAT in-context imitation; no training or finetuning |
| Capture device named | Intel RealSense 455, worn on the chest |
| Robot named | Sawyer with a Robotiq 2F-85 gripper |
| Dataset and code licence | Not stated in the sources read |
| Firsthand relationship | None |
11
R+X beside related entries on the site
| Site entry | What the entry states | How R+X differs (sources read) |
|---|---|---|
| MimicPlay | Latent plans from human play video guide a controller trained on few teleoperated demos | The R+X paper cites it as needing robot exploration or teleoperation; R+X does not |
| EgoMimic | Co-trains on egocentric human video with 3D hand tracking and on robot data | Not discussed in the paper; R+X trains and finetunes no model |
| Phantom | Trains policies from human video only by inpainting the arm and overlaying a rendered robot | Not discussed in the paper; R+X maps hand joints to gripper poses with heuristics |
| Point Policy | Learns robot policies from human videos using key points | Not discussed in the paper; R+X uses DINO-based keypoints as in-context input |
12
What R+X consumes beside what the Firsthand site states
| Aspect | R+X (sources read) | What the Firsthand site states |
|---|---|---|
| Human video | One long unlabelled first-person video, chest-worn camera | Egocentric capture, custom collection to a written spec |
| Depth | RGB-D camera | Depth derived from a calibrated stereo rig |
| Hand pose | 21 joints estimated by HaMeR | 3D hand pose at 21 joints per hand |
| Language | Command given at deployment only | One task instruction per episode |
| Robot-side data | 6-DoF gripper poses generated by the method | Robot trajectories are not Firsthand deliverables |
13
Frequently asked questions
- What is R+X?
- Per the paper (arXiv 2407.12957), R+X is a framework that learns robot skills from long, unlabelled first-person human video: it retrieves clips for a language command, then executes with in-context imitation. It is third-party research.
- How does R+X learn robot skills from long unlabelled first-person video?
- A vision language model finds the clips matching the command. Hand joints and keypoints from those clips become examples for in-context imitation, which outputs gripper poses.
- Does R+X train or finetune a model on the video?
- No. The authors state it needs no training or finetuning on the retrieved videos, so commanded skills run immediately.
- What did the authors report for R+X?
- They report an average success of 0.7 over 12 tasks against 0.55 and 0.53 for two baselines. Firsthand has not verified these figures.
- Does an R+X-style approach need stereo depth or IMUs?
- The paper describes a single RGB-D camera with no extrinsics calibration and does not state a need for stereo depth or IMUs.
- Was R+X trained on Firsthand data?
- No. Firsthand is not affiliated with the authors and no named model was trained on Firsthand data. Robot trajectories and action labels are not Firsthand deliverables.
- How can I request a fast, on-demand custom egocentric dataset for this research?
- Request a custom dataset with your tasks and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep.
Related terms