Glossary
What Is XSkill (Cross Embodiment Skill Discovery)?
Last reviewed: November 2026
Short answer
XSkill is a framework from Columbia University and J.P. Morgan AI Research (arXiv 2307.09955, CoRL 2023) that learns shared skill prototypes for humans and robots from unlabelled human and robot videos, then transfers them with a skill-conditioned diffusion policy and composes them to perform an unseen task from one human prompt video. Firsthand is not affiliated.
01
What is XSkill, and what problem does it address?
XSkill is named in XSkill: Cross Embodiment Skill Discovery (arXiv 2307.09955) by Mengda Xu, Zhenjia Xu, Cheng Chi, Manuela Veloso and Shuran Song of Columbia University, J.P. Morgan AI Research and Carnegie Mellon University, presented at the 7th Conference on Robot Learning (CoRL 2023). The authors note that human demonstration videos are a widely available data source, but extracting reusable skills from unstructured human video is hard because of the large embodiment difference and unobserved action parameters. They formulate the task as cross-embodiment skill discovery.
02
Where does Firsthand fit next to XSkill?
Firsthand is not affiliated with the XSkill authors. No named model, XSkill included, was trained on Firsthand data, and Firsthand does not claim to use any hardware or software named in the sources: a UR5 robot with a WSG50 gripper, Realsense D415 cameras, a 3Dconnexion SpaceMouse and diffusion policy. Firsthand captures real human egocentric video, custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license, with one task instruction per episode. Robot teleoperation data, robot trajectories, retargeted actions, action labels, skill prototypes, pretrained representations, embeddings, simulation environments, synthetic data and language annotations beyond that instruction are not Firsthand deliverables.
03
How does XSkill discover shared skill prototypes?
Per the paper, XSkill has three phases: Discover, Transfer and Compose. Discover uses a human demonstration dataset and a robot teleoperation dataset. Both are unsegmented and unaligned, and the authors state that no task labels or correspondence between embodiments are needed in training. A temporal skill encoder, a vision backbone plus a transformer encoder, maps video clips into one shared skill space.
That space is learned with K learnable skill prototypes, a normalized linear layer, using self-supervised clustering similar to SwAV. Sinkhorn-Knopp clustering inside single-embodiment batches keeps every prototype in use, so humans and robots share them, and a time contrastive loss captures how a skill unfolds over time. The authors argue off-the-shelf vision features are often too sensitive to how the agent or scene looks.
04
How does one human prompt video become robot actions?
In Transfer, the authors train a skill-conditioned visuomotor policy on the robot teleoperation data only, building on diffusion policy. It takes the skill representation and the robot state and outputs an action sequence. In Compose, a single human prompt video of an unseen task is mapped into the skill space, giving a sequence of skills that works as a task plan.
A Skill Alignment Transformer then compares the robot’s current observation with that plan, so the robot can tell which skills are already done. The authors say this helps with execution speed differences and failed attempts. They also state the method does not use reinforcement learning in the loop.
05
What data does XSkill learn from?
The authors built two cross-embodiment benchmarks. The simulated one uses a Franka Kitchen environment with a visually different sphere agent in place of the human. The real-world one has four sub-tasks: opening the oven, grasping a cloth, closing a drawer and turning on a light. It holds 175 human and 175 teleoperation demonstrations, recorded on a UR5 robot station with two Realsense D415 cameras (wrist and side view), with a 3Dconnexion SpaceMouse for teleoperation. Firsthand does not claim to use any of this hardware.
The video is the authors’ own; no public human video dataset is named as a source. The abstract says the benchmark, code and qualitative results are on the project website. Any dataset or code licence is Not stated in the sources read.
06
What did the authors report, and what are the limits?
With cross-embodiment prompts on unseen tasks, the authors report 70.2% success in simulation and 60% in the real world, above their baselines. They report that in the real world, cross-embodiment prompts cost around 5% against same-embodiment prompts on seen tasks. Removing the prototype loss, or the Skill Alignment Transformer, degrades results as the agent moves faster.
Stated limit: efficacy is still limited by the diversity of the robot teleoperation data, and the robot struggled with transitions absent from it. These are the authors’ figures; Firsthand has not verified them.
07
How does XSkill compare with related entries on this site?
The table sets XSkill beside three site entries and the general term. Unlike R+X, XSkill trains models, and it needs robot teleoperation data as well as human video.
08
What would an XSkill-style approach need from custom collection?
Per the sources read, it needs unlabelled human demonstration video plus a robot teleoperation dataset, a robot and a policy training stack, all the buyer’s own. The datasets need no task labels or paired demonstrations, so a per-episode instruction is not a stated requirement. Firsthand covers the human video side only.
This is custom collection: request on-demand collection and confirm it on a Spec Pilot, with a median of 48 hours to a first validated batch where coverage is deep and three to four weeks where it is thin.
09
XSkill at a glance, as stated in the paper and project page
| Item | What the sources state |
|---|---|
| Paper | XSkill: Cross Embodiment Skill Discovery (arXiv 2307.09955) |
| Authors | Mengda Xu, Zhenjia Xu, Cheng Chi, Manuela Veloso, Shuran Song |
| Institutions and venue | Columbia University, J.P. Morgan AI Research, Carnegie Mellon University; CoRL 2023 |
| Discover | Skill prototypes learned from unlabelled human and robot videos |
| Transfer | Skill-conditioned diffusion policy trained on robot teleoperation data |
| Compose | One human prompt video, aligned by a Skill Alignment Transformer |
| Robot hardware named | UR5 with a WSG50 gripper, two Realsense D415 cameras |
| Dataset and code licence | Not stated in the sources read |
| Firsthand relationship | None |
11
XSkill beside related entries on the site
| Site entry | What the entry states | How XSkill differs (sources read) |
|---|---|---|
| MimicPlay | Latent plans from human play video guide a controller trained on few teleoperated demos | The XSkill paper says MimicPlay aims to minimize robot demo collection, while XSkill learns a cross-embodiment representation to reduce reliance on robot demonstrations at inference |
| X-Sim | Real-to-sim-to-real framework that learns robot policies from action-less human videos using object motion | Not discussed in the paper; XSkill uses skill prototypes and a diffusion policy, with no simulator in the method |
| R+X | Retrieves clips for a command and conditions in-context imitation on them, with no training | Not discussed in the paper; XSkill trains a skill encoder and policy |
| Cross-embodiment learning | General term for training on data from multiple embodiments, including human demonstrations | XSkill is one published method for it, built around shared skill prototypes |
12
What XSkill consumes beside what the Firsthand site states
| Aspect | XSkill (sources read) | What the Firsthand site states |
|---|---|---|
| Human video | Unlabelled human demonstration video, RGB only | Egocentric capture, custom collection to a written spec |
| Depth | Not used by the method | Depth derived from a calibrated stereo rig |
| Hand pose | Not used by the method | 3D hand pose at 21 joints per hand |
| Language | None in training; the task is given by a human prompt video | One task instruction per episode |
| Robot-side data | Teleoperation dataset with proprioception and end-effector actions | Robot trajectories are not Firsthand deliverables |
13
Frequently asked questions
- What is XSkill?
- Per the paper (arXiv 2307.09955), XSkill is an imitation learning framework that discovers skill prototypes from unlabelled human and robot videos, transfers them with a diffusion policy and composes them for unseen tasks. It is third-party research.
- What are shared skill prototypes for humans and robots?
- They are learnable anchor vectors in a shared embedding space. Human and robot clips with similar action effects map to the same prototypes, which aligns skills across embodiments.
- Does XSkill learn from human video alone?
- No. The method also uses a robot teleoperation dataset, which trains the skill-conditioned policy. The human video defines the new task as a single prompt video.
- What did the authors report for XSkill?
- They report 70.2% success in simulation and 60% in the real world on unseen tasks with cross-embodiment prompts, above their baselines. Firsthand has not verified these figures.
- Does an XSkill-style approach need depth, hand pose or language labels?
- The paper describes RGB video and states no task labels or paired demonstrations are needed. It does not state a need for depth, hand pose or language.
- Was XSkill trained on Firsthand data?
- No. Firsthand is not affiliated with the authors and no named model was trained on Firsthand data. Robot teleoperation data and skill prototypes are not Firsthand deliverables.
- How can I request a fast, on-demand custom egocentric dataset for this research?
- Request a custom dataset with your tasks and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep.
Related terms
Sources
- XSkill: Cross Embodiment Skill Discovery (arXiv 2307.09955)
- XSkill project page (official, Columbia University)
- XSkill on OpenReview (CoRL 2023, linked from the project page)
- Diffusion Policy: Visuomotor Policy Learning via Action Diffusion (arXiv 2303.04137)
- MimicPlay: Long-Horizon Imitation Learning by Watching Human Play (arXiv 2302.12422)