Glossary
What Is X-Sim (Cross-Embodiment Learning via Real-to-Sim-to-Real)?
Last reviewed: November 2026
Short answer
X-Sim is the real-to-sim-to-real framework in “X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Real” (arXiv 2505.07096, Cornell University). It rebuilds a simulation from an RGBD human video, turns tracked object motion into object-centric rewards, trains a reinforcement learning policy there, then distills it into an image-based policy for a real robot. Third-party; Firsthand is not affiliated.
01
What is X-Sim, and what problem does it address?
X-Sim is named in X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Real (arXiv 2505.07096) by Prithwish Dan, Kushal Kedia, Angela Chao, Edward W. Duan, Maximus A. Pace, Wei-Chiu Ma and Sanjiban Choudhury of Cornell University; the project page lists an oral at CoRL 2025. The authors say human videos are a scalable way to train robot manipulation policies but lack the action labels that standard imitation learning needs, and that approaches mapping human motion to robot actions often fail when the embodiments differ significantly. Their answer is to use the motion of the objects, not the human hand, as the transferable signal. See also the site’s embodiment gap, sim-to-real gap and cross-embodiment learning entries.
02
Where does Firsthand fit next to X-Sim?
Firsthand is not affiliated with the X-Sim authors. No named model, X-Sim included, was trained on Firsthand data, and Firsthand does not claim to use any hardware or software named in the paper: it uses a 7-DOF Franka arm, a ZED 2 stereo camera for the human RGBD videos, a phone scanning app and the ManiSkill simulator. Firsthand captures real human egocentric video, custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license, with one task instruction per episode. Object meshes, object pose tracks, object-centric rewards, simulation environments or digital twins, Gaussian splat reconstructions, reinforcement learning policies, synthetic image-action data, paired real and simulated images, robot trajectories, action labels, reward labels, pretrained representations, embeddings and other language annotations are not Firsthand deliverables.
03
How does X-Sim turn a human video into a robot policy?
Per the paper, X-Sim has three stages. In real-to-sim, objects are scanned with a phone app to get meshes, FoundationPose tracks each object through the RGBD human video starting from a Segment Anything mask, and 2D Gaussian Splatting reconstructs the environment from multi-view images, which is placed in the ManiSkill simulator. Physical properties are left at default values.
In the second stage the tracked object poses define a dense object-centric reward that encourages the robot to move each object toward its next pose in the human trajectory. A privileged-state policy is trained with PPO, then rolled out under randomized object starts, camera viewpoints and lighting, keeping only successful trajectories, to build a synthetic image-action dataset. In the third stage an image-conditioned Diffusion Policy is trained on that dataset. The paper adds an online domain adaptation step: real rollouts are replayed in simulation to make paired real and simulated images, and a contrastive loss aligns the encoder’s embeddings.
04
What did the authors report, and what are the limits?
The authors evaluate five tasks in two real environments, Kitchen and Tabletop, on a 7-DOF Franka arm, with human videos recorded by a ZED 2 stereo camera, over 10 trials per task. They report an average 30% gain in task progress over hand-tracking and sim-to-real baselines, matching behavior cloning with 10x less data collection time, and generalization to new camera viewpoints. Calibration adds an average 8% in task progress. On Mustard Place they report 90% success from 1 minute of human video against 70% from 10 minutes of robot demonstrations. These are the authors’ figures; Firsthand has not verified them.
The paper names limits: the pipeline needs an object mesh to track, is limited to rigid objects, needs a 3D scan of the environment, and uses default physics parameters. The authors say applying X-Sim to unstructured in-the-wild internet video remains an open challenge.
05
How does X-Sim compare with related entries on this site?
The last tables set X-Sim beside site entries for related methods and concepts. In the paper’s framing, X-Sim differs from hand-retargeting and hand-masking approaches because it learns robot-specific strategies in simulation that reproduce object motion, rather than copying the human hand.
06
What data does an X-Sim-style approach need, and what does Firsthand state?
Per the paper, it needs RGBD human demonstration videos, object meshes for tracking, multi-view images of the environment, a simulator and a robot to deploy on. The authors record their own videos, so a dataset licence is Not stated in the sources read. That is a note about format, not a compatibility claim: object tracking, scene reconstruction, simulation and the robot are the buyer’s own.
No off-the-shelf dataset for this is stated here, so this is custom collection: request on-demand collection and confirm it on a Spec Pilot, with a median of 48 hours to a first validated batch where coverage is deep and three to four weeks where it is thin.
07
X-Sim at a glance, as stated in the paper and project page
| Item | What the sources state |
|---|---|
| Paper | X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Real (arXiv 2505.07096) |
| Authors | Prithwish Dan, Kushal Kedia, Angela Chao, Edward W. Duan, Maximus A. Pace, Wei-Chiu Ma, Sanjiban Choudhury (Cornell University) |
| Venue | Oral at CoRL 2025, per the project page |
| Input | An action-less RGBD human video per task, plus scans of the environment and objects |
| Reward signal | Object-centric: tracked object poses from the human video |
| Policy | Image-conditioned Diffusion Policy distilled from a PPO policy trained in simulation |
| Hardware and tools named | 7-DOF Franka arm, ZED 2 stereo camera, ManiSkill simulator, FoundationPose, 2D Gaussian Splatting |
| Robot teleoperation data | Not required, per the authors |
| Code | The abstract points to the project page for code and videos |
| Dataset licence | Not stated in the sources read |
| Firsthand relationship | None |
09
X-Sim pipeline, as stated in the paper
| Stage | What the paper states |
|---|---|
| Object tracking | Object meshes from a phone scan; FoundationPose tracks position and rotation through the RGBD video |
| Environment | 2D Gaussian Splatting reconstruction from multi-view images, transferred into ManiSkill |
| Reward | Dense object-centric reward toward the next object pose in the human trajectory |
| RL | Privileged-state policy trained with PPO; only successful rollouts are kept |
| Synthetic data | Image-action pairs rendered under randomized object starts, viewpoints and lighting |
| Policy | Image-conditioned Diffusion Policy, no privileged state at inference |
| Adaptation | Real rollouts replayed in simulation; a contrastive loss aligns paired real and simulated images |
10
X-Sim beside related entries on the site
| Site entry | What the entry states | How X-Sim differs (sources read) |
|---|---|---|
| Sim-to-real gap | The mismatch between a simulator’s approximations and real-world physics, lighting and material behavior | X-Sim narrows it with randomized rendering and an online calibration step on paired real and simulated images |
| Embodiment gap | The mismatch between a human body and a robot’s hardware that keeps human demonstration data from transferring directly | X-Sim crosses it with object motion as the signal, so the robot need not copy human hand motion |
| Cross-embodiment learning | Training one policy on data from multiple embodiments, including human demonstrations | X-Sim uses human video only, with no robot teleoperation data, per the authors |
| Phantom | Trains robot policies from human video only, by inpainting the arm and overlaying a rendered robot | The paper uses Phantom’s hand-mask approach as a baseline; X-Sim learns in simulation instead |
| Motion Tracks (MT-π) | Uses 2D image trajectories as actions for human hands and robots | The paper cites it among methods needing robot teleoperation data for low-level control; X-Sim does not |
| Zero-shot manipulation from passive human videos | Predicts human hand trajectories from a scene image, learned from passive video | The paper lists it among hand-retargeting methods; X-Sim rewards object motion instead |
11
What X-Sim consumes beside what the Firsthand site states
| Aspect | X-Sim (sources read) | What the Firsthand site states |
|---|---|---|
| Source video | Authors’ own RGBD human videos recorded with a ZED 2 stereo camera | Egocentric capture, custom collection to a written spec |
| Depth | RGBD video used for object pose tracking | Depth derived from a calibrated stereo rig |
| Hand signals | Not used for RL training, per the paper | 3D hand pose at 21 joints per hand |
| Language | Not described as an input | One task instruction per episode |
| Simulation-side data | Object tracks, rewards, simulation scenes and synthetic image-action data produced by the authors | Object tracks, rewards, simulation environments, synthetic data and action labels are not Firsthand deliverables |
12
Frequently asked questions
- What is X-Sim?
- Per the paper (arXiv 2505.07096), X-Sim is a real-to-sim-to-real framework that learns image-based robot policies from action-less human videos by tracking object motion and matching it in simulation. It is third-party research.
- What are object-centric rewards in X-Sim?
- They are rewards computed from the tracked poses of the objects in the human video. The reward encourages the robot to move each object toward its next position and orientation, even if the robot uses very different actions from the human.
- Does X-Sim need robot teleoperation data?
- No, per the authors. The policy is trained from simulation rollouts and then adapted online with paired real and simulated images, without teleoperation data. It does need object meshes and a 3D scan of the environment.
- What did the authors report for X-Sim?
- They report an average 30% gain in task progress over hand-tracking and sim-to-real baselines on five tasks in two environments, and matching behavior cloning with 10x less data collection time. Firsthand has not verified these figures.
- Was X-Sim trained on Firsthand data?
- No. Firsthand is not affiliated with the authors and no named model was trained on Firsthand data. Object tracks, rewards, simulation environments, synthetic data and action labels are not Firsthand deliverables.
- How does X-Sim differ from retargeting human hand motion?
- Retargeting copies hand motion to the robot end effector, which the authors say often fails when embodiments differ. X-Sim instead lets reinforcement learning in simulation find robot-specific actions that reproduce the same object motion.
- How can I request a custom egocentric dataset for human-to-robot learning?
- Request a custom dataset with your objects, interactions and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep.
Related terms
Sources
- X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Real (arXiv 2505.07096)
- X-Sim (official project page, linked from the paper)
- FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects (arXiv 2312.08344)
- 2D Gaussian Splatting for Geometrically Accurate Radiance Fields (arXiv 2403.17888)
- ManiSkill (official GitHub repository)
- Diffusion Policy: Visuomotor Policy Learning via Action Diffusion (arXiv 2303.04137)
- Phantom: Training Robots Without Robots Using Only Human Videos (arXiv 2503.00779)