Why Does Embodied AI Need Real-World Video?

Real-world video, in the context of embodied AI, means footage recorded by a camera physically present in an actual environment — a real room, a real kitchen, a real warehouse aisle — rather than rendered by a graphics engine or scraped from a platform built for human viewing. For robot policies, vision-language-action (VLA) models, and world models, the distinction matters because the video is standing in for the sensory experience the deployed system will actually have: real light sources, real surface textures, real object weight and friction, and a real body (human or robot) acting within that scene.

On this site, real-world video specifically means egocentric capture: a camera worn at head or chest height, paired with synchronized depth, IMU, and 3D hand or body pose on a shared clock. That pairing is what turns a single video stream into training data a model can learn to act from, rather than footage it can only watch.

Simulation is cheap to scale: once a scene is built, a physics engine can render many lighting conditions, object permutations, and camera angles automatically, with no scheduling, travel, or recruiting involved. That is a real advantage for coverage. But every simulated frame is an approximation — of physics, of material behavior, of how light falls across a surface — and a policy trained only on those approximations can perform well inside the simulator while still failing on the real dynamics it never saw. This is the sim-to-real gap, and it is a known limitation of simulation-only training rather than something a bigger simulator necessarily solves.

The gap shows up most clearly in exactly the signals embodied AI depends on for manipulation: contact timing, grasp-level hand pose, and how an object’s weight and friction actually behave when a hand closes around it. A simulator’s hand and physics model has to be designed to reproduce those dynamics, and how well it does so varies by simulator and task — real capture records them directly from the event instead of depending on how faithfully a model reproduces them.

For the full side-by-side on where simulation helps and where it falls short against real egocentric capture, see egocentric data vs synthetic data for robot learning.

Internet video is abundant, free to view, and shows a wide range of real human activity — which makes it tempting as a training source. But it was never captured for robot training: it is monocular RGB with no depth channel, no 3D hand pose, and no synchronized multi-camera clock, because none of those were needed for a human viewer watching a single feed. Any depth or hand-pose signal a team wants from it has to be estimated after the fact from 2D pixels, which introduces exactly the kind of noise that directly captured labels do not carry.

Internet video also carries consent and licensing uncertainty that real-world captured data is built to avoid from day one: footage scraped from public platforms was not collected with training-data consent from the people appearing in it, and its rights to reuse for commercial model training are frequently unclear or contested. A model trained on it without resolving those questions inherits that uncertainty.

The practical effect is that internet video can teach a model what a task looks like — the appearance and rough motion of an activity — but not what it takes to act: the depth, contact timing, and hand-pose precision a manipulation policy actually trains on.

Real-world egocentric capture, recorded with a worn camera rig hardware-triggered onto one clock, gives a model several things at once that neither simulation nor internet video provides together: active-stereo depth captured directly alongside RGB rather than estimated after the fact, 3D hand pose and contact events recorded live during the event, and head-plus-wrist camera geometry that keeps the hand-eye relationship consistent throughout a task. None of that has to be inferred or approximated — it is measured at the moment it happens.

It also captures real clutter, real lighting variation, and real human variation — different bodies, grips, paces, and recovery behaviors — because the capture happens in an actual environment with real people, not a scene a designer chose to model or a clip someone else happened to upload. A policy trained on that variation meets fewer surprises at deployment than one trained only on a simulator’s or an uploader’s choices about what to include.

Neither simulation nor internet video is useless — each is good at something real-world capture is not. Simulation is the cheapest way to generate broad scene and object permutations once a scene is built, which makes it useful for early-stage coverage or for rare configurations that would be expensive to stage for real. Internet video, despite its missing signals, can still support broad task-appearance pretraining — teaching a model what a wide range of activities generally look like — before a narrower, purpose-captured dataset refines it on the depth, hand-pose, and contact detail that internet footage cannot provide.

The pattern most robot-learning programs land on is a blend rather than a single source: simulation or internet video for cheap, broad coverage, and real-world egocentric capture to ground the policy in the exact physics, lighting, and hand-contact dynamics it will actually meet, particularly for the failure cases and edge conditions the other two sources are least likely to reproduce faithfully.

A dataset built only from clean, successful task completions teaches a model what success looks like, but not what to do when a grasp slips, an object occludes a view, lighting changes mid-task, or a recovery is needed — situations a deployed system will encounter regardless of whether training data covered them. Real-world capture is where those cases naturally occur or can be deliberately specced, because they are drawn from real attempts rather than a simulator’s or an uploader’s choice of what to show.

Diversity across scenes, tasks, objects, and participants is also a documented lever for how well a policy generalizes, not a byproduct of simply collecting more hours — GEN-0’s official write-up frames data diversity and quality as mattering as much as sheer data volume. A real-world collection that covers a narrow set of rooms, objects, or people is cheaper to capture but risks teaching a model a narrower slice of the world than it will actually face.

Not every piece of real-world video is equally useful to a model — the signals it carries matter as much as the fact that it was recorded in a real place. A capture that only has a single RGB stream, with no depth, no hand pose, and no synchronized clock across cameras, still lacks most of what simulation and internet video are also missing, even though it is technically "real."

EgoScale’s own published pipeline illustrates the tradeoff directly: it estimates action labels from SLAM and hand-pose estimation after the fact rather than capturing them live, which is a documented source of label noise compared with labels recorded at capture time.

Commissioning real-world egocentric capture starts with a written spec: the task categories, scenes, object set, and participant diversity the dataset needs, plus which streams each episode must carry and what sync and calibration tolerance is required. That spec is what turns "we need real-world video" into something a capture team can actually plan against and quote, rather than a vague request.

Custom egocentric collection on this site is on-demand rather than fixed to a pre-built catalog: a buyer writes or works out a spec, a small Spec Pilot batch is captured and validated against it first, and scaled capture follows once the pilot confirms the setup. Every delivered hour is checked against the spec before it is billed, action labels and hand pose are captured live rather than estimated afterward, consent is recorded per participant with faces and other identifying details detected and blurred, and the data ships under a perpetual, buyer-owned commercial license in formats robot-learning pipelines already read — RLDS, LeRobot, and WebDataset.

For the full mechanics of how a custom collection is specced, piloted, and scaled, see how custom robotics data collection works and robot pretraining data.