Training Data for Vision-Language-Action (VLA) Models

A vision-language-action (VLA) model takes in a visual scene and a language instruction and outputs an action — a robot policy, gripper command, or hand trajectory. That mapping only exists in the training data if each episode pairs the visual frames with an instruction and the action stream that carried it out, all on a timeline where the three are provably simultaneous. A video-caption pair with no action stream, or an action log with no instruction, is missing one leg of the triple.

Egocentric video puts the camera at the embodiment carrying out the instruction — a head or chest mount, or a wrist camera on the hand itself — so the viewpoint matches what a robot or embodied agent will actually see at inference time. That keeps hands, tools, and the instructed object in frame at the moment of contact, rather than off to the side of a fixed third-person shot.

• The viewpoint transfers directly to deployment, instead of requiring the policy to relearn geometry from a different camera angle
• Hands and the instructed object stay in frame at the moment of contact, rather than occluded from an outside view
• Head, wrist, and chest mounts can run in the same episode on one synchronized clock alongside the instruction and action log

The instruction on its own only labels intent. To train the action head, the episode needs the action itself — 3D hand pose, gripper or contact state, or robot action logs — hardware-triggered onto the same clock as the video, so a given frame, a given instruction, and a given action are provably simultaneous rather than loosely time-stamped.

Every sensor in the rig hardware-triggers onto one PTP clock domain, so video, hand pose, and action logs share a timebase rather than being merely co-recorded — the measured residual across delivered episodes runs a median of 1.12 ms and a p99 of 1.94 ms, with a 2.0 ms reject ceiling. That sync is what makes a frame, its instruction, and the action inside it usable as a training triple.

The full sync measurement, verified against real episode data, is in "How to measure cross-stream sync error" below.

A VLA dataset is collected against a written spec that names the tasks, the instruction set, and the condition budget, rather than pulled from a fixed catalog — so reach, timeline, and price are quoted against what the model actually needs. A first validated batch ships in a median of 48 hours where coverage is deep.