Now recruiting capture teams worldwide. Get paid to record first-person video of real work.

Apply to capture

Glossary

What Is X-Sim (Cross-Embodiment Learning via Real-to-Sim-to-Real)?

X-Sim is a published real-to-sim-to-real framework that learns robot policies from action-less human videos using object motion. Third-party; Firsthand is not affiliated.

Last reviewed: November 2026

Short answer

X-Sim is the real-to-sim-to-real framework in “X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Real” (arXiv 2505.07096, Cornell University). It rebuilds a simulation from an RGBD human video, turns tracked object motion into object-centric rewards, trains a reinforcement learning policy there, then distills it into an image-based policy for a real robot. Third-party; Firsthand is not affiliated.

01

What is X-Sim, and what problem does it address?

X-Sim is named in X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Real (arXiv 2505.07096) by Prithwish Dan, Kushal Kedia, Angela Chao, Edward W. Duan, Maximus A. Pace, Wei-Chiu Ma and Sanjiban Choudhury of Cornell University; the project page lists an oral at CoRL 2025. The authors say human videos are a scalable way to train robot manipulation policies but lack the action labels that standard imitation learning needs, and that approaches mapping human motion to robot actions often fail when the embodiments differ significantly. Their answer is to use the motion of the objects, not the human hand, as the transferable signal. See also the site’s embodiment gap, sim-to-real gap and cross-embodiment learning entries.

02

Where does Firsthand fit next to X-Sim?

Firsthand is not affiliated with the X-Sim authors. No named model, X-Sim included, was trained on Firsthand data, and Firsthand does not claim to use any hardware or software named in the paper: it uses a 7-DOF Franka arm, a ZED 2 stereo camera for the human RGBD videos, a phone scanning app and the ManiSkill simulator. Firsthand captures real human egocentric video, custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license, with one task instruction per episode. Object meshes, object pose tracks, object-centric rewards, simulation environments or digital twins, Gaussian splat reconstructions, reinforcement learning policies, synthetic image-action data, paired real and simulated images, robot trajectories, action labels, reward labels, pretrained representations, embeddings and other language annotations are not Firsthand deliverables.

03

How does X-Sim turn a human video into a robot policy?

Per the paper, X-Sim has three stages. In real-to-sim, objects are scanned with a phone app to get meshes, FoundationPose tracks each object through the RGBD human video starting from a Segment Anything mask, and 2D Gaussian Splatting reconstructs the environment from multi-view images, which is placed in the ManiSkill simulator. Physical properties are left at default values.

In the second stage the tracked object poses define a dense object-centric reward that encourages the robot to move each object toward its next pose in the human trajectory. A privileged-state policy is trained with PPO, then rolled out under randomized object starts, camera viewpoints and lighting, keeping only successful trajectories, to build a synthetic image-action dataset. In the third stage an image-conditioned Diffusion Policy is trained on that dataset. The paper adds an online domain adaptation step: real rollouts are replayed in simulation to make paired real and simulated images, and a contrastive loss aligns the encoder’s embeddings.

04

What did the authors report, and what are the limits?

The authors evaluate five tasks in two real environments, Kitchen and Tabletop, on a 7-DOF Franka arm, with human videos recorded by a ZED 2 stereo camera, over 10 trials per task. They report an average 30% gain in task progress over hand-tracking and sim-to-real baselines, matching behavior cloning with 10x less data collection time, and generalization to new camera viewpoints. Calibration adds an average 8% in task progress. On Mustard Place they report 90% success from 1 minute of human video against 70% from 10 minutes of robot demonstrations. These are the authors’ figures; Firsthand has not verified them.

The paper names limits: the pipeline needs an object mesh to track, is limited to rigid objects, needs a 3D scan of the environment, and uses default physics parameters. The authors say applying X-Sim to unstructured in-the-wild internet video remains an open challenge.

05

How does X-Sim compare with related entries on this site?

The last tables set X-Sim beside site entries for related methods and concepts. In the paper’s framing, X-Sim differs from hand-retargeting and hand-masking approaches because it learns robot-specific strategies in simulation that reproduce object motion, rather than copying the human hand.

06

What data does an X-Sim-style approach need, and what does Firsthand state?

Per the paper, it needs RGBD human demonstration videos, object meshes for tracking, multi-view images of the environment, a simulator and a robot to deploy on. The authors record their own videos, so a dataset licence is Not stated in the sources read. That is a note about format, not a compatibility claim: object tracking, scene reconstruction, simulation and the robot are the buyer’s own.

No off-the-shelf dataset for this is stated here, so this is custom collection: request on-demand collection and confirm it on a Spec Pilot, with a median of 48 hours to a first validated batch where coverage is deep and three to four weeks where it is thin.

07

X-Sim at a glance, as stated in the paper and project page

X-Sim at a glance, as stated in the paper and project page
ItemWhat the sources state
PaperX-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Real (arXiv 2505.07096)
AuthorsPrithwish Dan, Kushal Kedia, Angela Chao, Edward W. Duan, Maximus A. Pace, Wei-Chiu Ma, Sanjiban Choudhury (Cornell University)
VenueOral at CoRL 2025, per the project page
InputAn action-less RGBD human video per task, plus scans of the environment and objects
Reward signalObject-centric: tracked object poses from the human video
PolicyImage-conditioned Diffusion Policy distilled from a PPO policy trained in simulation
Hardware and tools named7-DOF Franka arm, ZED 2 stereo camera, ManiSkill simulator, FoundationPose, 2D Gaussian Splatting
Robot teleoperation dataNot required, per the authors
CodeThe abstract points to the project page for code and videos
Dataset licenceNot stated in the sources read
Firsthand relationshipNone

09

X-Sim pipeline, as stated in the paper

X-Sim pipeline, as stated in the paper
StageWhat the paper states
Object trackingObject meshes from a phone scan; FoundationPose tracks position and rotation through the RGBD video
Environment2D Gaussian Splatting reconstruction from multi-view images, transferred into ManiSkill
RewardDense object-centric reward toward the next object pose in the human trajectory
RLPrivileged-state policy trained with PPO; only successful rollouts are kept
Synthetic dataImage-action pairs rendered under randomized object starts, viewpoints and lighting
PolicyImage-conditioned Diffusion Policy, no privileged state at inference
AdaptationReal rollouts replayed in simulation; a contrastive loss aligns paired real and simulated images

10

X-Sim beside related entries on the site

X-Sim beside related entries on the site
Site entryWhat the entry statesHow X-Sim differs (sources read)
Sim-to-real gapThe mismatch between a simulator’s approximations and real-world physics, lighting and material behaviorX-Sim narrows it with randomized rendering and an online calibration step on paired real and simulated images
Embodiment gapThe mismatch between a human body and a robot’s hardware that keeps human demonstration data from transferring directlyX-Sim crosses it with object motion as the signal, so the robot need not copy human hand motion
Cross-embodiment learningTraining one policy on data from multiple embodiments, including human demonstrationsX-Sim uses human video only, with no robot teleoperation data, per the authors
PhantomTrains robot policies from human video only, by inpainting the arm and overlaying a rendered robotThe paper uses Phantom’s hand-mask approach as a baseline; X-Sim learns in simulation instead
Motion Tracks (MT-π)Uses 2D image trajectories as actions for human hands and robotsThe paper cites it among methods needing robot teleoperation data for low-level control; X-Sim does not
Zero-shot manipulation from passive human videosPredicts human hand trajectories from a scene image, learned from passive videoThe paper lists it among hand-retargeting methods; X-Sim rewards object motion instead

11

What X-Sim consumes beside what the Firsthand site states

What X-Sim consumes beside what the Firsthand site states
AspectX-Sim (sources read)What the Firsthand site states
Source videoAuthors’ own RGBD human videos recorded with a ZED 2 stereo cameraEgocentric capture, custom collection to a written spec
DepthRGBD video used for object pose trackingDepth derived from a calibrated stereo rig
Hand signalsNot used for RL training, per the paper3D hand pose at 21 joints per hand
LanguageNot described as an inputOne task instruction per episode
Simulation-side dataObject tracks, rewards, simulation scenes and synthetic image-action data produced by the authorsObject tracks, rewards, simulation environments, synthetic data and action labels are not Firsthand deliverables

12

Frequently asked questions

What is X-Sim?
Per the paper (arXiv 2505.07096), X-Sim is a real-to-sim-to-real framework that learns image-based robot policies from action-less human videos by tracking object motion and matching it in simulation. It is third-party research.
What are object-centric rewards in X-Sim?
They are rewards computed from the tracked poses of the objects in the human video. The reward encourages the robot to move each object toward its next position and orientation, even if the robot uses very different actions from the human.
Does X-Sim need robot teleoperation data?
No, per the authors. The policy is trained from simulation rollouts and then adapted online with paired real and simulated images, without teleoperation data. It does need object meshes and a 3D scan of the environment.
What did the authors report for X-Sim?
They report an average 30% gain in task progress over hand-tracking and sim-to-real baselines on five tasks in two environments, and matching behavior cloning with 10x less data collection time. Firsthand has not verified these figures.
Was X-Sim trained on Firsthand data?
No. Firsthand is not affiliated with the authors and no named model was trained on Firsthand data. Object tracks, rewards, simulation environments, synthetic data and action labels are not Firsthand deliverables.
How does X-Sim differ from retargeting human hand motion?
Retargeting copies hand motion to the robot end effector, which the authors say often fails when embodiments differ. X-Sim instead lets reinforcement learning in simulation find robot-specific actions that reproduce the same object motion.
How can I request a custom egocentric dataset for human-to-robot learning?
Request a custom dataset with your objects, interactions and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep.