Now recruiting capture teams worldwide. Get paid to record first-person video of real work.

Apply to capture

Glossary

What Is Masquerade (Learning from In-the-Wild Human Videos via Data-Editing)?

Masquerade edits in-the-wild egocentric human videos into robotized demonstrations to train robot policies. Third-party; Firsthand is not affiliated.

Last reviewed: November 2026

Short answer

Masquerade is the method in “Masquerade: Learning from In-the-wild Human Videos using Data-Editing” (arXiv 2508.09976, Stanford University). It edits egocentric human videos into robotized demonstrations by estimating hand poses, inpainting the arms and overlaying a rendered bimanual robot, then trains a policy on the edited clips plus a few robot demonstrations. Third-party; Firsthand is not affiliated.

01

What is Masquerade, and what problem does it address?

Masquerade is named in Masquerade: Learning from In-the-wild Human Videos using Data-Editing (arXiv 2508.09976) by Marion Lepert, Jiaying Fang and Jeannette Bohg of Stanford University; the project page lists ICRA 2026. The authors say robot datasets are orders of magnitude smaller than language and vision datasets, and that human videos lack action labels and look different from a robot. They extend the data-editing pipeline of Phantom, shown only on carefully collected single-hand videos from a fixed camera, to in-the-wild videos. See also the site’s embodiment gap, Phantom and cross-embodiment learning entries.

02

Where does Firsthand fit next to Masquerade?

Firsthand is not affiliated with the Masquerade authors. No named model, Masquerade included, was trained on Firsthand data, and Firsthand does not claim to use any hardware named in the paper: its robot setup is two Kinova Gen3 arms with Robotiq 2F-85 grippers and a ZED mini camera, and the human videos come from the existing Epic Kitchens dataset. Firsthand captures real human egocentric video, custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license, with one task instruction per episode. Edited or inpainted video, robot overlays, rendered robot poses, 2D robot keypoints, retargeted end-effector poses, robot trajectories, action labels, reward labels, co-training mixtures, pretrained representations, embeddings, teleoperation data, paired human-robot video, synthetic data and other language annotations are not Firsthand deliverables.

03

How does Masquerade turn human video into robot demonstrations?

Per the paper, each clip goes through three steps. Hands are localized with Epic Kitchens annotations and 21 keypoints per hand are estimated with HaMeR, then mapped to an end-effector pose with position, orientation and gripper opening width, and temporally smoothed. The human arms are segmented with Detectron2 and SAM 2 and removed with E2FGVI inpainting. A virtual bimanual robot whose end effectors follow the poses is rendered with known camera intrinsics and extrinsics and composited into the frame.

Frames with excessive camera motion or invalid actions are filtered out. The authors say many overlays stay imperfect: the retargeting cannot handle every dexterous grasp, and without depth, robot pixels sometimes appear over scene objects.

04

What does the policy learn from the edited clips?

Because the videos have no depth, the authors use projected 2D keypoint locations as labels for an auxiliary loss rather than feeding them into policy learning. A ViT-Base encoder, initialized from ImageNet weights, is pretrained to predict future 2D robot keypoints on the edited clips, conditioned on each clip’s language description embedded with DistilBERT.

It is then co-trained with a Diffusion Policy head on 50 robot demonstrations per task from a single scene, with the auxiliary loss kept on. The authors inpaint a rendered robot over the real robot so the model always sees a rendered robot, and collected those demonstrations with an Oculus headset.

05

What did the authors report, and what are the limits?

The authors train on 10K Epic Kitchens clips (675K frames) and test on three long-horizon bimanual kitchen tasks, each in three unseen scenes. The abstract reports outperforming baselines by 5-6x; the paper reports an average gain of 62 percentage points (12% to 74%). Ablations without the overlay or without co-training drop sharply. On Stack Pots, success is 2%, 26%, 47% and 68% at 0%, 10%, 50% and 100% of the edited videos, over 25 rollouts each. These are the authors’ figures; Firsthand has not verified them.

The paper names limits: hand pose estimators struggle with fast motion and occlusion, depth is missing, camera motion forces frame filtering because the robot is stationary, and mapping dexterous grasps to a parallel-jaw gripper is imperfect.

06

How does Masquerade compare with Phantom, EgoMimic, Point Policy and related entries?

The last table sets Masquerade beside five site entries. Per the paper, Masquerade combines visual editing with co-training and applies both to uncurated in-the-wild video, where earlier methods used curated demonstrations.

07

What data does a Masquerade-style approach need, and what does Firsthand state?

Per the paper, it needs a large set of egocentric human videos with language descriptions and hand locations, a hand pose estimator, segmentation and inpainting models, a renderable bimanual robot, known camera intrinsics and extrinsics, and a small robot dataset per task. The Epic Kitchens licence is Not stated in the sources read, so check the dataset’s own terms. That is a note about format, not a compatibility claim: pose estimation, inpainting, rendering and the robot are the buyer’s own.

No off-the-shelf dataset for this is stated here, so this is custom collection: request on-demand collection and confirm it on a Spec Pilot, with a median of 48 hours to a first validated batch where coverage is deep and three to four weeks where it is thin.

08

Masquerade at a glance, as stated in the paper and project page

Masquerade at a glance, as stated in the paper and project page
ItemWhat the sources state
PaperMasquerade: Learning from In-the-wild Human Videos using Data-Editing (arXiv 2508.09976)
AuthorsMarion Lepert, Jiaying Fang, Jeannette Bohg (Stanford University)
VenueICRA 2026, per the project page
Human video sourceEpic Kitchens: 10K clips, 675K frames (authors’ counts)
Robot data50 demonstrations per task, one scene (authors’ count)
PolicyViT-Base encoder with a Diffusion Policy head
Robot hardware namedTwo Kinova Gen3 arms, Robotiq 2F-85 grippers, ZED mini camera
CodeNot stated in the sources read
Dataset licenceNot stated in the sources read
Firsthand relationshipNone

10

Masquerade pipeline, as stated in the paper

Masquerade pipeline, as stated in the paper
StageWhat the paper states
Hand pose21 keypoints per hand from HaMeR, mapped to position, orientation and gripper width, then smoothed
InpaintingArms segmented with Detectron2 and SAM 2, removed with E2FGVI
Robot overlayRendered bimanual robot follows the poses using known camera intrinsics and extrinsics
LabelsProjected 2D keypoints for the next H waypoints, corrected for camera motion with a homography
PretrainingViT-Base predicts future 2D robot keypoints, conditioned on per-clip language
Co-trainingDiffusion Policy head on 50 robot demonstrations per task, auxiliary loss kept

11

Masquerade beside related entries on the site

Masquerade beside related entries on the site
Site entryWhat the entry statesHow Masquerade differs (sources read)
PhantomTrains robot policies from human video only, by inpainting the arm and overlaying a rendered robotMasquerade extends Phantom’s pipeline to in-the-wild egocentric video and adds co-training with robot demonstrations
EgoMimicCo-trains robot manipulation policies on egocentric human video with 3D hand tracking and on robot dataThe paper groups EgoMimic with methods using curated demonstrations; Masquerade edits uncurated videos
Humanoid Policy ~ Human PolicyPairs the PH2D dataset with the HAT policy to co-train humanoids on egocentric human dataThe paper cites PH2D among curated co-training methods; Masquerade overlays a robot on the video
Point PolicyLearns robot policies from human videos using key pointsThe paper lists it among point-flow methods on curated video; Masquerade edits the pixels instead
Motion Tracks (MT-π)Uses 2D image trajectories as actions for human hands and robotsAlso listed among point-flow methods; Masquerade uses 2D keypoints only as an auxiliary label

12

What Masquerade consumes beside what the Firsthand site states

What Masquerade consumes beside what the Firsthand site states
AspectMasquerade (sources read)What the Firsthand site states
Source videoExisting in-the-wild egocentric videos from Epic KitchensEgocentric capture, custom collection to a written spec
DepthNone; the authors say the videos lack depth dataDepth derived from a calibrated stereo rig
Hand signals21 keypoints per hand estimated from monocular frames3D hand pose at 21 joints per hand
LanguageA natural language description per clipOne task instruction per episode
Robot-side dataEdited video, overlays and labels produced by the authorsRobot overlays, edited video, robot trajectories and action labels are not Firsthand deliverables

13

Frequently asked questions

What is Masquerade?
Per the paper (arXiv 2508.09976), Masquerade is a method that edits in-the-wild egocentric human videos into robotized demonstrations and learns a robot policy from them. It is third-party research.
How does Masquerade edit human videos into robot demonstrations?
It estimates 3D hand poses, inpaints the human arms and overlays a rendered bimanual robot that follows the recovered end-effector trajectories, so each clip looks like a robot performing the task.
Does Masquerade need robot data?
Yes. The authors co-train on 50 robot demonstrations per task from one scene, and their ablations say both the robot overlay and co-training are indispensable. This differs from Phantom, which uses no robot data.
What did the authors report for Masquerade?
The abstract reports outperforming baselines by 5-6x on three bimanual kitchen tasks in unseen scenes, and the paper reports an average of 62 percentage points (12% to 74%). Firsthand has not verified these figures.
Was Masquerade trained on Firsthand data?
No. Firsthand is not affiliated with the authors and no named model was trained on Firsthand data. Edited video, robot overlays, 2D robot keypoints, robot trajectories and action labels are not Firsthand deliverables.
How does Masquerade differ from Phantom?
The authors say they extend Phantom’s data-editing pipeline, shown on curated single-hand videos with a fixed camera, to in-the-wild egocentric videos, and pair it with co-training on a small robot dataset.
How can I request a custom egocentric dataset for human-to-robot learning?
Request a custom dataset with your objects, interactions and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep.