Glossary
What Is the Sim-to-Real Gap?
Last reviewed: September 2026
Short answer
The sim-to-real gap is the mismatch between how a simulator approximates physics, lighting, and material behavior and how those things actually behave in the real world. A policy trained only on synthetic, simulation-rendered data can perform well in simulation but fail when those approximations don’t match reality — particularly for hand-contact dynamics, which depend entirely on how well the simulator’s hand and physics model reproduces the real event.
01
Why it matters
Simulation scales scene and object variation cheaply, but every synthetic frame carries the simulator’s approximations forward into the policy. Real egocentric capture matches deployment physics, lighting, and contact dynamics exactly because it is real, which is why most robot-learning programs blend both rather than training on synthetic data alone.
02
How Firsthand handles it
Firsthand captures real egocentric data — head and wrist cameras, active-stereo depth, and IMU, hardware-triggered on one clock — not simulation-rendered data, collected fast, on demand, and to a written spec. It pairs with a program’s existing synthetic data to close the coverage-versus-grounding tradeoff simulation alone can’t solve.
03
Where the sim-to-real gap shows up, and what real egocentric capture adds
| Where the gap shows up | What simulation approximates | What real egocentric capture provides instead |
|---|---|---|
| Visuals | Rendered lighting, textures, and materials | Real-world lighting, reflectance, and material appearance, because it is real |
| Physics and contact | A simulator’s hand and physics model | 3D hand pose, metric depth, and action segments captured directly from the real event |
| Sensing | Simulated camera and depth noise characteristics | Active-stereo depth and camera streams with real sensor noise, hardware-synchronized on one clock |
| Human behavior and failure cases | Modeled failure modes the simulator’s designer thought to include | Real failure cases — transit, occlusion, low light, dropped objects, recovery — with real sensor noise and physics |
05
Frequently asked questions
- What is the sim-to-real gap?
- It is the mismatch between how a simulator approximates physics, lighting, and material behavior and how those things actually behave in the real world. A policy trained only on synthetic data can perform well in simulation but fail when those approximations don’t match reality.
- What is sim2real?
- Sim2real (or sim-to-real transfer) is the general problem of getting a policy trained in simulation to work in the real world despite the sim-to-real gap.
- Why is synthetic data cheaper but riskier than real egocentric data?
- A simulator can render many scene, lighting, and object permutations automatically once it is built, but every rendered frame carries its approximations forward. Real egocentric capture matches deployment conditions exactly but needs a real person, a real site, and a worn camera rig for every session.
- Does real egocentric data eliminate the sim-to-real gap?
- It avoids the gap entirely for the frames it captures, because those frames are real. It doesn’t replace simulation’s ability to scale scene and object variation cheaply — most programs blend both.
- Where does the sim-to-real gap show up most in manipulation?
- Hand and contact fidelity is usually the hardest part to close, since grasp-level manipulation depends on precise hand pose and contact timing that a simulator can only approximate through its hand and physics model.