Now recruiting capture teams worldwide. Get paid to record first-person video of real work.

Apply to capture

Glossary

What Is AR2-D2 (Training a Robot Without a Robot)?

AR2-D2 is a published system that collects robot demonstrations with a mobile AR app, with no real robot. Third-party; Firsthand is not affiliated.

Last reviewed: November 2026

Short answer

AR2-D2 is the system in “AR2-D2: Training a Robot Without a Robot” (arXiv 2306.13818, University of Washington). It is a mobile AR app that people use to record themselves manipulating an object while it captures the data needed to train a real robot, with no specially trained operators and no real robot. Third-party; Firsthand is not affiliated.

01

What is AR2-D2, and what problem does it address?

AR2-D2 is named in AR2-D2: Training a Robot Without a Robot (arXiv 2306.13818) by Jiafei Duan, Yi Ru Wang, Mohit Shridhar, Dieter Fox and Ranjay Krishna; the project page lists CoRL 2023 and the University of Washington, NVIDIA and the Allen Institute for AI. The authors say demonstrations are usually collected by training people to use specialized controllers that tele-operate robots, which limits collection to a small number of objects. Their answer is a capture app that lets anyone demonstrate with any object and turns the recording into robot-style training data. See also the site’s Phantom, embodiment gap and cross-embodiment learning entries.

02

Where does Firsthand fit next to AR2-D2?

Firsthand is not affiliated with the AR2-D2 authors. No named model, AR2-D2 included, was trained on Firsthand data, and Firsthand does not claim to use any hardware or software named in the paper: it names a Franka Panda robot arm model, the Unity Engine with AR Foundation, Segment Anything, E2FGVI and Perceiver-Actor. Firsthand captures real human egocentric video, custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license, with one task instruction per episode. AR-rendered robot demonstrations, virtual robot overlays, hand-removed videos, 3D voxelized scene representations, robot trajectories, action labels, simulation environments, pretrained representations, embeddings and other language annotations are not Firsthand deliverables.

03

How does AR2-D2 turn a recording into robot training data?

Per the project page, AR2-D2 is a mobile app built on the Unity Engine and AR Foundation. It reads camera data, including intrinsic and extrinsic values, and depth from the device itself, and uses the device’s built-in LiDAR to place an AR robot into the real scene. A Franka Panda arm model is projected into the space. The app detects the user’s 2D hand pose with a built-in human pose detection algorithm and combines it with the depth map to reconstruct a 3D hand pose, tracked at 30 frames per second so the AR robot end-effector can mimic it.

Given a language instruction such as “Pick up the plastic bowl”, hired users record a demonstration. The authors then segment out and remove the human hand with Segment Anything, fill the gap with the E2FGVI video in-painting technique, and render the AR robot arm moving to the key points taken from the hand, so the video looks as if an AR robot had manipulated the object. Because depth is available, they also build a 3D voxelized scene representation to train agents such as Perceiver-Actor.

04

What did the authors report, and what are the limits?

The authors report that data collected with AR2-D2 enables training behavior cloning agents on real objects, and that training with their AR data is as effective as training with real-world robot demonstrations. Their user study with 10 subjects found AR2-D2 faster than four other conventional interfaces for collecting demonstrations, in simulation and in the real world, and found it intuitive and requiring no training. The project page describes demonstrations across three manipulation tasks, press, pick-up and push, on 9 personalized objects, and reports that 5 demonstrations and 3,000 iterations, about 10 minutes of training, gave the best result when fine-tuning on the deployment environment. These are the authors’ figures; Firsthand has not verified them.

The authors name one limit: AR2-D2 cannot capture physical interaction fidelity, because a human hand cannot replicate the exact force a real robot applies. They argue that for most objects in quasi-static manipulation the difference does not impede transfer.

05

How does AR2-D2 compare with related entries on this site?

The tables below set AR2-D2 beside site entries for related methods and concepts. AR2-D2 was posted to arXiv in June 2023. Its core idea, replacing the human hand with a rendered robot arm so a policy sees robot-like video, is also the idea the site describes for Phantom and Masquerade; what the sources read single out for AR2-D2 is that the capture app records depth and tracks the hand while the person demonstrates.

06

What data does an AR2-D2-style approach need, and what does Firsthand state?

Per the sources read, it needs a mobile device with a camera, depth and LiDAR running the authors’ app, people to record demonstrations from language instructions, and a real robot to deploy on. The authors recorded their own demonstrations, so a dataset licence is Not stated in the sources read. That is a note about format, not a compatibility claim: the AR rendering, in-painting and training pipeline are the buyer’s own.

No off-the-shelf dataset for this is stated here, so this is custom collection: request on-demand collection and confirm it on a Spec Pilot, with a median of 48 hours to a first validated batch where coverage is deep and three to four weeks where it is thin.

07

AR2-D2 at a glance, as stated in the paper and project page

AR2-D2 at a glance, as stated in the paper and project page
ItemWhat the sources state
PaperAR2-D2: Training a Robot Without a Robot (arXiv 2306.13818)
AuthorsJiafei Duan, Yi Ru Wang, Mohit Shridhar, Dieter Fox, Ranjay Krishna
VenueCoRL 2023, per the project page
Capture deviceA mobile AR app with built-in camera, depth and LiDAR
Real robot during collectionNot required, per the authors
Hand tracking2D hand pose plus depth gives 3D hand pose, tracked at 30 frames per second
Tools namedUnity Engine, AR Foundation, Segment Anything, E2FGVI, Perceiver-Actor
Robot model namedFranka Panda arm, rendered as an AR overlay
Dataset licenceNot stated in the sources read
Firsthand relationshipNone

09

AR2-D2 data pipeline, as stated in the paper and project page

AR2-D2 data pipeline, as stated in the paper and project page
StageWhat the sources state
CaptureUser records a video of themselves manipulating an object in the app, given a language instruction
SensingCamera intrinsics and extrinsics and depth come from the device; LiDAR helps place the AR robot
Hand pose2D hand pose combined with the depth map to reconstruct 3D hand pose
Hand removalHand segmented with Segment Anything, gap filled with E2FGVI
Robot overlayAR robot arm rendered moving to key points taken from the hand
Training dataProcessed video for image-based imitation learning, plus a 3D voxelized scene for Perceiver-Actor

10

AR2-D2 beside related entries on the site

AR2-D2 beside related entries on the site
Site entryWhat the entry statesHow AR2-D2 differs (sources read)
PhantomTrains robot policies from human video only, by inpainting the arm and overlaying a rendered robotShares the inpaint-and-overlay idea; AR2-D2 uses a capture app that records depth and hand pose during the demonstration
MasqueradeEdits in-the-wild egocentric human videos into robotized demonstrationsAR2-D2 describes users hired to record with its app, not edits to in-the-wild video
X-SimRebuilds a simulation from an RGBD human video and trains a policy thereThe AR2-D2 sources describe AR-rendered robot video as the training data, not a rebuilt simulation
Embodiment gapThe mismatch between a human body and a robot’s hardware that keeps human demonstration data from transferring directlyAR2-D2 addresses it by showing the policy an AR robot arm in place of the hand
Cross-embodiment learningTraining one policy on data from multiple embodiments, including human demonstrationsAR2-D2 produces robot-style demonstrations from a human recording; the authors report no real robot is needed to collect them

11

What AR2-D2 consumes beside what the Firsthand site states

What AR2-D2 consumes beside what the Firsthand site states
AspectAR2-D2 (sources read)What the Firsthand site states
Source videoRecordings made in the authors’ mobile AR appEgocentric capture, custom collection to a written spec
DepthDepth from the mobile deviceDepth derived from a calibrated stereo rig
Hand signals3D hand pose reconstructed from 2D pose and depth at 30 frames per second3D hand pose at 21 joints per hand
LanguageA language instruction describes each taskOne task instruction per episode
Robot-side dataAR robot overlays, hand-removed video and voxelized scenes produced by the authorsAR robot demonstrations, overlays, voxel scenes, robot trajectories and action labels are not Firsthand deliverables

12

Frequently asked questions

What is AR2-D2?
Per the paper (arXiv 2306.13818), AR2-D2 is a system for collecting robot demonstrations in the form of a mobile AR app. People record themselves manipulating any object, and the app captures the data needed to train a real robot. It is third-party research.
Does AR2-D2 need a real robot to collect data?
No, per the authors. Collection needs no real robot and no specially trained operators; a real robot is used only to deploy the trained agent.
How does AR2-D2 make a human video look like a robot demonstration?
The authors segment out and remove the human hand, fill the gap with video in-painting, and render an AR robot arm moving to the key points taken from the hand. The result looks as if an AR robot manipulated the object.
What did the authors report for AR2-D2?
They report that training with their AR data is as effective as training with real-world robot demonstrations, and that a 10-subject user study found it faster than four other interfaces. Firsthand has not verified these figures.
Was AR2-D2 trained on Firsthand data?
No. Firsthand is not affiliated with the authors and no named model was trained on Firsthand data. AR-rendered robot demonstrations, overlays, voxel scenes and action labels are not Firsthand deliverables.
How does AR2-D2 relate to Phantom and Masquerade?
All three show a policy a rendered robot in place of the human hand. Per the sources read, AR2-D2 does so through a capture app that records depth and hand pose during the demonstration.
How can I request a custom egocentric dataset for human-to-robot learning?
Request a custom dataset with your objects, interactions and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep.