Glossary
What Is Masquerade (Learning from In-the-Wild Human Videos via Data-Editing)?
Last reviewed: November 2026
Short answer
Masquerade is the method in “Masquerade: Learning from In-the-wild Human Videos using Data-Editing” (arXiv 2508.09976, Stanford University). It edits egocentric human videos into robotized demonstrations by estimating hand poses, inpainting the arms and overlaying a rendered bimanual robot, then trains a policy on the edited clips plus a few robot demonstrations. Third-party; Firsthand is not affiliated.
01
What is Masquerade, and what problem does it address?
Masquerade is named in Masquerade: Learning from In-the-wild Human Videos using Data-Editing (arXiv 2508.09976) by Marion Lepert, Jiaying Fang and Jeannette Bohg of Stanford University; the project page lists ICRA 2026. The authors say robot datasets are orders of magnitude smaller than language and vision datasets, and that human videos lack action labels and look different from a robot. They extend the data-editing pipeline of Phantom, shown only on carefully collected single-hand videos from a fixed camera, to in-the-wild videos. See also the site’s embodiment gap, Phantom and cross-embodiment learning entries.
02
Where does Firsthand fit next to Masquerade?
Firsthand is not affiliated with the Masquerade authors. No named model, Masquerade included, was trained on Firsthand data, and Firsthand does not claim to use any hardware named in the paper: its robot setup is two Kinova Gen3 arms with Robotiq 2F-85 grippers and a ZED mini camera, and the human videos come from the existing Epic Kitchens dataset. Firsthand captures real human egocentric video, custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license, with one task instruction per episode. Edited or inpainted video, robot overlays, rendered robot poses, 2D robot keypoints, retargeted end-effector poses, robot trajectories, action labels, reward labels, co-training mixtures, pretrained representations, embeddings, teleoperation data, paired human-robot video, synthetic data and other language annotations are not Firsthand deliverables.
03
How does Masquerade turn human video into robot demonstrations?
Per the paper, each clip goes through three steps. Hands are localized with Epic Kitchens annotations and 21 keypoints per hand are estimated with HaMeR, then mapped to an end-effector pose with position, orientation and gripper opening width, and temporally smoothed. The human arms are segmented with Detectron2 and SAM 2 and removed with E2FGVI inpainting. A virtual bimanual robot whose end effectors follow the poses is rendered with known camera intrinsics and extrinsics and composited into the frame.
Frames with excessive camera motion or invalid actions are filtered out. The authors say many overlays stay imperfect: the retargeting cannot handle every dexterous grasp, and without depth, robot pixels sometimes appear over scene objects.
04
What does the policy learn from the edited clips?
Because the videos have no depth, the authors use projected 2D keypoint locations as labels for an auxiliary loss rather than feeding them into policy learning. A ViT-Base encoder, initialized from ImageNet weights, is pretrained to predict future 2D robot keypoints on the edited clips, conditioned on each clip’s language description embedded with DistilBERT.
It is then co-trained with a Diffusion Policy head on 50 robot demonstrations per task from a single scene, with the auxiliary loss kept on. The authors inpaint a rendered robot over the real robot so the model always sees a rendered robot, and collected those demonstrations with an Oculus headset.
05
What did the authors report, and what are the limits?
The authors train on 10K Epic Kitchens clips (675K frames) and test on three long-horizon bimanual kitchen tasks, each in three unseen scenes. The abstract reports outperforming baselines by 5-6x; the paper reports an average gain of 62 percentage points (12% to 74%). Ablations without the overlay or without co-training drop sharply. On Stack Pots, success is 2%, 26%, 47% and 68% at 0%, 10%, 50% and 100% of the edited videos, over 25 rollouts each. These are the authors’ figures; Firsthand has not verified them.
The paper names limits: hand pose estimators struggle with fast motion and occlusion, depth is missing, camera motion forces frame filtering because the robot is stationary, and mapping dexterous grasps to a parallel-jaw gripper is imperfect.
06
How does Masquerade compare with Phantom, EgoMimic, Point Policy and related entries?
The last table sets Masquerade beside five site entries. Per the paper, Masquerade combines visual editing with co-training and applies both to uncurated in-the-wild video, where earlier methods used curated demonstrations.
07
What data does a Masquerade-style approach need, and what does Firsthand state?
Per the paper, it needs a large set of egocentric human videos with language descriptions and hand locations, a hand pose estimator, segmentation and inpainting models, a renderable bimanual robot, known camera intrinsics and extrinsics, and a small robot dataset per task. The Epic Kitchens licence is Not stated in the sources read, so check the dataset’s own terms. That is a note about format, not a compatibility claim: pose estimation, inpainting, rendering and the robot are the buyer’s own.
No off-the-shelf dataset for this is stated here, so this is custom collection: request on-demand collection and confirm it on a Spec Pilot, with a median of 48 hours to a first validated batch where coverage is deep and three to four weeks where it is thin.
08
Masquerade at a glance, as stated in the paper and project page
| Item | What the sources state |
|---|---|
| Paper | Masquerade: Learning from In-the-wild Human Videos using Data-Editing (arXiv 2508.09976) |
| Authors | Marion Lepert, Jiaying Fang, Jeannette Bohg (Stanford University) |
| Venue | ICRA 2026, per the project page |
| Human video source | Epic Kitchens: 10K clips, 675K frames (authors’ counts) |
| Robot data | 50 demonstrations per task, one scene (authors’ count) |
| Policy | ViT-Base encoder with a Diffusion Policy head |
| Robot hardware named | Two Kinova Gen3 arms, Robotiq 2F-85 grippers, ZED mini camera |
| Code | Not stated in the sources read |
| Dataset licence | Not stated in the sources read |
| Firsthand relationship | None |
10
Masquerade pipeline, as stated in the paper
| Stage | What the paper states |
|---|---|
| Hand pose | 21 keypoints per hand from HaMeR, mapped to position, orientation and gripper width, then smoothed |
| Inpainting | Arms segmented with Detectron2 and SAM 2, removed with E2FGVI |
| Robot overlay | Rendered bimanual robot follows the poses using known camera intrinsics and extrinsics |
| Labels | Projected 2D keypoints for the next H waypoints, corrected for camera motion with a homography |
| Pretraining | ViT-Base predicts future 2D robot keypoints, conditioned on per-clip language |
| Co-training | Diffusion Policy head on 50 robot demonstrations per task, auxiliary loss kept |
11
Masquerade beside related entries on the site
| Site entry | What the entry states | How Masquerade differs (sources read) |
|---|---|---|
| Phantom | Trains robot policies from human video only, by inpainting the arm and overlaying a rendered robot | Masquerade extends Phantom’s pipeline to in-the-wild egocentric video and adds co-training with robot demonstrations |
| EgoMimic | Co-trains robot manipulation policies on egocentric human video with 3D hand tracking and on robot data | The paper groups EgoMimic with methods using curated demonstrations; Masquerade edits uncurated videos |
| Humanoid Policy ~ Human Policy | Pairs the PH2D dataset with the HAT policy to co-train humanoids on egocentric human data | The paper cites PH2D among curated co-training methods; Masquerade overlays a robot on the video |
| Point Policy | Learns robot policies from human videos using key points | The paper lists it among point-flow methods on curated video; Masquerade edits the pixels instead |
| Motion Tracks (MT-π) | Uses 2D image trajectories as actions for human hands and robots | Also listed among point-flow methods; Masquerade uses 2D keypoints only as an auxiliary label |
12
What Masquerade consumes beside what the Firsthand site states
| Aspect | Masquerade (sources read) | What the Firsthand site states |
|---|---|---|
| Source video | Existing in-the-wild egocentric videos from Epic Kitchens | Egocentric capture, custom collection to a written spec |
| Depth | None; the authors say the videos lack depth data | Depth derived from a calibrated stereo rig |
| Hand signals | 21 keypoints per hand estimated from monocular frames | 3D hand pose at 21 joints per hand |
| Language | A natural language description per clip | One task instruction per episode |
| Robot-side data | Edited video, overlays and labels produced by the authors | Robot overlays, edited video, robot trajectories and action labels are not Firsthand deliverables |
13
Frequently asked questions
- What is Masquerade?
- Per the paper (arXiv 2508.09976), Masquerade is a method that edits in-the-wild egocentric human videos into robotized demonstrations and learns a robot policy from them. It is third-party research.
- How does Masquerade edit human videos into robot demonstrations?
- It estimates 3D hand poses, inpaints the human arms and overlays a rendered bimanual robot that follows the recovered end-effector trajectories, so each clip looks like a robot performing the task.
- Does Masquerade need robot data?
- Yes. The authors co-train on 50 robot demonstrations per task from one scene, and their ablations say both the robot overlay and co-training are indispensable. This differs from Phantom, which uses no robot data.
- What did the authors report for Masquerade?
- The abstract reports outperforming baselines by 5-6x on three bimanual kitchen tasks in unseen scenes, and the paper reports an average of 62 percentage points (12% to 74%). Firsthand has not verified these figures.
- Was Masquerade trained on Firsthand data?
- No. Firsthand is not affiliated with the authors and no named model was trained on Firsthand data. Edited video, robot overlays, 2D robot keypoints, robot trajectories and action labels are not Firsthand deliverables.
- How does Masquerade differ from Phantom?
- The authors say they extend Phantom’s data-editing pipeline, shown on curated single-hand videos with a fixed camera, to in-the-wild egocentric videos, and pair it with co-training on a small robot dataset.
- How can I request a custom egocentric dataset for human-to-robot learning?
- Request a custom dataset with your objects, interactions and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep.
Related terms
Sources
- Masquerade: Learning from In-the-wild Human Videos using Data-Editing (arXiv 2508.09976)
- Masquerade (official project page, linked from the paper)
- Phantom: Training Robots Without Robots Using Only Human Videos (arXiv 2503.00779)
- EPIC-KITCHENS dataset (official site)
- Reconstructing Hands in 3D with Transformers (HaMeR, arXiv 2312.05251)
- SAM 2: Segment Anything in Images and Videos (arXiv 2408.00714)
- Diffusion Policy: Visuomotor Policy Learning via Action Diffusion (arXiv 2303.04137)