Now recruiting capture teams worldwide. Get paid to record first-person video of real work.

Apply to capture

Glossary

What Is R+X (Retrieval and Execution from Everyday Human Videos)?

R+X learns robot skills from long unlabelled first-person video: a VLM retrieves clips for a command, then in-context imitation executes. Third-party.

Last reviewed: November 2026

Short answer

R+X, Retrieval and Execution, is a framework from Imperial College London (arXiv 2407.12957) that teaches robots skills from long, unlabelled first-person videos of humans doing everyday tasks. Given a language command, it retrieves relevant clips with a vision language model, then conditions in-context imitation learning on them, with no training on those clips. Firsthand is not affiliated.

01

What is R+X, and what problem does it address?

R+X is named in R+X: Retrieval and Execution from Everyday Human Videos (arXiv 2407.12957) by Georgios Papagiannis, Norman Di Palo, Pietro Vitiello and Edward Johns of the Robot Learning Lab at Imperial College London, submitted on 17 July 2024 and published at ICRA 2025. The authors study robot skills from long, unlabelled, first-person videos of a person going about daily life. They remove assumptions of earlier work: videos aligned with robot videos, language labels, or demonstrations on robot or motion-capture hardware.

02

Where does Firsthand fit next to R+X?

Firsthand is not affiliated with the R+X authors. No named model, R+X included, was trained on Firsthand data, and Firsthand does not claim to use any hardware or software named in the sources: a chest-worn Intel RealSense 455, Gemini Pro 1.5 Flash, DINO, HaMeR, KAT, and a Sawyer robot with a Robotiq 2F-85 gripper. Firsthand captures real human egocentric video, custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license, with one task instruction per episode. Robot trajectories, retargeted actions, action labels, hand pose priors, 3D object models, simulation environments, co-training mixtures, pretrained representations, embeddings, synthetic data and language annotations beyond that instruction are not Firsthand deliverables.

03

How does R+X turn a long unlabelled video into a robot skill?

Per the paper, R+X takes three inputs: one long unlabelled human video, a language command and the robot’s current RGB-D observation. It outputs a trajectory of 6-DoF gripper poses. In retrieval, a vision language model, Gemini Pro 1.5 Flash, is prompted with the video and asked for the start and end seconds where the task happens, returning shorter clips.

In execution, each clip is reduced to sparse 3D data: visual keypoints from DINO descriptors and a 21-joint hand trajectory from HaMeR. These become examples for Keypoint Action Tokens (KAT), an in-context imitation method built on off-the-shelf language models, which predicts hand joints for the live scene. The authors state there is no training or finetuning, so skills run immediately.

04

What video does R+X learn from?

The authors recorded the human video with an Intel RealSense 455 worn on the chest, in many rooms and buildings, then used HaMeR to drop frames without visible hands. Because the camera moves, each clip is stabilised with CLIPSeg segmentation and TAPIR point tracking. The project page also reports chest, head and third-person views. The paper says R+X needs a single RGB-D camera and no extrinsics calibration.

The authors recorded their own video; no public human video dataset is named. Any video or code licence is Not stated in the sources read.

05

How do hand joints become gripper actions?

The authors apply heuristics by interaction type. For grasping they align the gripper fingertips with the index and thumb tips; for pressing and pushing they align points on a closed gripper with index finger joints. Gemini picks the heuristic from the command. They target a Robotiq 2F-85 on a Sawyer robot with a RealSense 415 head camera, and note a human-like dexterous hand would not need them. Firsthand does not claim to use any of this hardware.

06

What did the authors report, and what are the limits?

On 12 everyday tasks, with 10 evaluations per task, the authors report an average success of 0.7 for R+X against 0.55 for R3M-DiffLang and 0.53 for Octo, both fine-tuned baselines trained on Gemini-captioned clips. They also report strong performance with as few as 10 retrieved clips, and no drop on earlier tasks when new ones are added.

Stated limits: hand-pose errors limit precise tasks, keypoints can bottleneck with many similar objects, and there is a delay of a few seconds. These are the authors’ figures; Firsthand has not verified them.

07

How does R+X compare with related entries on this site?

The table sets R+X beside four site entries. R+X retrieves clips for each command and conditions in-context, rather than training a policy.

08

What would an R+X-style approach need from custom collection?

Per the sources read, it needs a long first-person video that includes hands, an RGB-D view, a vision language model and a robot, all the buyer’s own. The paper needs no labels or calibration, so a per-episode instruction is not a stated requirement.

This is custom collection: request on-demand collection and confirm it on a Spec Pilot, with a median of 48 hours to a first validated batch where coverage is deep and three to four weeks where it is thin.

09

R+X at a glance, as stated in the paper and project page

R+X at a glance, as stated in the paper and project page
ItemWhat the sources state
PaperR+X: Retrieval and Execution from Everyday Human Videos (arXiv 2407.12957)
AuthorsGeorgios Papagiannis, Norman Di Palo, Pietro Vitiello, Edward Johns
Institution and venueImperial College London; ICRA 2025
RetrievalGemini Pro 1.5 Flash returns start and end seconds of relevant clips
ExecutionKAT in-context imitation; no training or finetuning
Capture device namedIntel RealSense 455, worn on the chest
Robot namedSawyer with a Robotiq 2F-85 gripper
Dataset and code licenceNot stated in the sources read
Firsthand relationshipNone

11

R+X beside related entries on the site

R+X beside related entries on the site
Site entryWhat the entry statesHow R+X differs (sources read)
MimicPlayLatent plans from human play video guide a controller trained on few teleoperated demosThe R+X paper cites it as needing robot exploration or teleoperation; R+X does not
EgoMimicCo-trains on egocentric human video with 3D hand tracking and on robot dataNot discussed in the paper; R+X trains and finetunes no model
PhantomTrains policies from human video only by inpainting the arm and overlaying a rendered robotNot discussed in the paper; R+X maps hand joints to gripper poses with heuristics
Point PolicyLearns robot policies from human videos using key pointsNot discussed in the paper; R+X uses DINO-based keypoints as in-context input

12

What R+X consumes beside what the Firsthand site states

What R+X consumes beside what the Firsthand site states
AspectR+X (sources read)What the Firsthand site states
Human videoOne long unlabelled first-person video, chest-worn cameraEgocentric capture, custom collection to a written spec
DepthRGB-D cameraDepth derived from a calibrated stereo rig
Hand pose21 joints estimated by HaMeR3D hand pose at 21 joints per hand
LanguageCommand given at deployment onlyOne task instruction per episode
Robot-side data6-DoF gripper poses generated by the methodRobot trajectories are not Firsthand deliverables

13

Frequently asked questions

What is R+X?
Per the paper (arXiv 2407.12957), R+X is a framework that learns robot skills from long, unlabelled first-person human video: it retrieves clips for a language command, then executes with in-context imitation. It is third-party research.
How does R+X learn robot skills from long unlabelled first-person video?
A vision language model finds the clips matching the command. Hand joints and keypoints from those clips become examples for in-context imitation, which outputs gripper poses.
Does R+X train or finetune a model on the video?
No. The authors state it needs no training or finetuning on the retrieved videos, so commanded skills run immediately.
What did the authors report for R+X?
They report an average success of 0.7 over 12 tasks against 0.55 and 0.53 for two baselines. Firsthand has not verified these figures.
Does an R+X-style approach need stereo depth or IMUs?
The paper describes a single RGB-D camera with no extrinsics calibration and does not state a need for stereo depth or IMUs.
Was R+X trained on Firsthand data?
No. Firsthand is not affiliated with the authors and no named model was trained on Firsthand data. Robot trajectories and action labels are not Firsthand deliverables.
How can I request a fast, on-demand custom egocentric dataset for this research?
Request a custom dataset with your tasks and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep.