How Do You Collect Training Data for a VLA Model?

A vision-language-action (VLA) model learns from episodes that pair what the model sees with an instruction and an action. There is no single collection recipe. The five papers read for this guide assemble their data in different ways, and each describes its own approach in its own words. This guide reports those statements as the papers make them, attributes each one, and then describes how a custom collection is specified, captured, checked and delivered.

Collection starts with the training stage the data is for. The π0 paper describes a recipe in which "the model is first pre-trained on a very large and diverse corpus, and then fine-tuned on more narrow and more carefully curated data" (https://arxiv.org/abs/2410.24164). Pretraining data is collected for breadth. Fine-tuning data is collected for one task or a small set of tasks. Those are different collection jobs, and a spec for one is not a spec for the other.

The order below is the order the site describes for a custom collection. Each step is covered in its own section.

• Name the stage and the tasks. Say whether the data is for pretraining or fine-tuning, and list the tasks, scenes and objects it must cover.
• Write the skill spec. Put the task, the geometry that stays in frame, the accept criteria and the reject rule on paper before anyone captures anything.
• Choose the capture source. Published models used robot demonstrations, web data, simulation, teleoperation or human video. Each gives a different kind of action signal.
• Define the episode. Fix the observation streams, the action fields and the language field the episode must carry.
• Run a small first batch. Firsthand calls this a Spec Pilot: a small, fast batch that proves the spec and the accept criteria before volume is added.
• Validate and deliver. Every item clears the signed accept criteria, anonymization and consent checks, then ships in a training-ready format.

This guide covers where training data comes from and how a collection runs. For how much data to plan, see the guide on VLA dataset size. For the fields inside an episode, see the guide on VLA dataset format. For a capture-day checklist, see the egocentric data collection checklist. All three are linked under related pages.

The table reports, for each paper, how it says its training data was assembled. Cells use the paper’s own wording or a close attribution. Where a paper is silent on a point, the cell says "Not stated in the sources read". Nothing is converted, summed or ranked, because the papers count data in different units: robots and months, demonstrations, hours, trajectories and timesteps.

RT-2 and OpenVLA both start from robot demonstrations. RT-2 states how many robots and how many months produced its demonstrations, and OpenVLA states the size of the public dataset it trains on. Neither paper, in the passages read, says whether those demonstrations were collected by teleoperation, so this guide does not say so either.

π0 combines its own datasets with open-source data. It states the number of robot configurations and tasks behind its own data and the number of robots behind the open-source portion. GR00T N1 is the most explicit about mixing sources. Its pyramid puts web data and human video at the base, synthetic data in the middle and real-robot data at the top. It also says how it handles sources that carry no actions: "To train our model with action-less data sources, such as human videos and neural-generated videos, we learn a latent-action codebook and also use a trained inverse dynamics model (IDM) to infer pseudo-actions."

EgoScale is the one paper of the five that centres on egocentric human video. The paper says human data can be gathered at a scale that robot teleoperation cannot match: "Humans routinely perform dexterous manipulation across diverse objects, environments, and task variations at a scale that far exceeds what can be collected through robot teleoperation."

The papers do not agree on a single recipe, and none of them derives a general rule for which source to use. This guide does not derive one either. The pattern it can report is narrower: several of these papers combine more than one data source, and each source supplies a different kind of signal.

The site’s guide to VLA dataset format describes a VLA training episode as a timestamped sequence of observations, an action at each step and a language task instruction, stored in a format such as RLDS or LeRobot. That framing is the starting point here. Collection has to produce all three parts, or a clear plan for how your pipeline will produce the missing ones.

Firsthand collects first-person human video, so the fields it delivers sit on the observation and action side. The fields the site states are: head-camera and wrist-camera streams, 21-joint 3D hand pose with a per-joint visibility flag, wrist trajectories, contact events, action segments and a 200 Hz 6-DoF IMU. Where a task is depth-sensitive, the site states that calibrated stereo is added. The streams are hardware-triggered onto a single clock, so every field in an episode shares one timeline.

The language side is narrow by design. The only language field the site states for Firsthand capture is one task instruction per episode, held on the same clock as the rest of the episode. The site does not state narration, captions, dense language annotation or any other language data for Firsthand capture, so none is described here.

Every action field Firsthand delivers is continuous. If your model predicts discrete action tokens, or acts through a robot rather than a hand, chunking, tokenization and retargeting to your action space are steps your own pipeline performs after delivery. The site does not state a robot action space for Firsthand data, and it does not deliver robot actions.

Teleoperation and egocentric human video differ in where the action signal comes from. The table sets out what the two papers that describe each approach say. It is not a ranking, and the papers do not rank them.

The practical difference for a buyer is the mapping step. Human video contains no robot actions, so someone has to decide how hand motion maps to the actions your model learns. Firsthand delivers the human side and leaves that mapping to your pipeline. Firsthand captures first-person human video, not robot or teleoperation data, and it makes no claim to replace either.

The two are not exclusive. GR00T N1 and EgoScale both use human video alongside other sources rather than on its own. The choice for a given project depends on the tasks, the target robot and what your training code expects. The sources do not give a rule for it.

The site defines a skill spec as the written contract for a capture program. It names the exact task, the geometry that must stay in frame, an explicit hour budget per condition, the reject rule and one acceptance test a reviewer can apply. In the site’s words, sharp specs produce coverage and vague specs only produce hours. The full template is in the skill spec guide, linked under related pages.

For VLA data, three decisions belong in the spec before capture. First, the stage: pretraining data is planned for coverage across tasks, scenes and objects, while fine-tuning data is planned around one task or a few. Second, the unit: hours, episodes, trajectories and demonstrations are different counts, so the spec should fix one. Third, the language field: each episode carries one task instruction, so the spec should list the tasks the instructions will name.

The π0 paper says its pre-training phase uses "diverse language labels". Firsthand’s stated language field is narrower: one task instruction per episode. The site does not state how long an instruction is, how it is worded or how many distinct instructions a collection holds, so those points are Not stated. If your training code needs richer language than one instruction per episode, plan that annotation on your side or raise it in the brief before capture.

The spec also fixes the acceptance test. Because every item is checked against the signed accept criteria before delivery, a criterion that cannot be applied by a reviewer to a single episode is not yet a criterion. A small first batch, the Spec Pilot, is the way to find that out on real footage before volume is added.

On-demand does not mean lower standards. The site states that every item clears the signed accept criteria, anonymization and consent checks before delivery. Where coverage is already deep, the first validated batch arrives in a median of 48 hours from a signed spec. Where coverage is thin and contributors have to be recruited first, the site states three to four weeks. After the first batch, delivery is rolling, one validated batch at a time.

Delivery formats are RLDS, LeRobot, WebDataset and HDF5, and the data is buyer-owned. The table follows one collection from spec to delivery. The last column names an existing episode field where the site states one, and says "Not stated" where it does not.

Two limits apply. The site does not state an episode count, an hour count or a throughput figure for a custom VLA collection, so this guide gives none. And it states no result for any named VLA model trained on Firsthand data. Quality here means the item met the signed accept criteria, not that a model scored better on it.

Public mixtures such as Open X-Embodiment cover the robots and scenes their publishers collected. Custom collection makes sense when your tasks, objects, environments or label vocabulary fall outside that coverage, or when you need a narrow, filtered fine-tuning set faster than your own team can capture it.

• Your target tasks or objects are missing from the public data you can access, and a Spec Pilot would show it quickly.
• You need human-side egocentric episodes with hand pose, contact events and action segments on one clock, and you can map hand motion to your own action space.
• You want the unit, the accept criteria and the delivery format written down before capture starts, so the dataset loads into the training code you already run.
• You need to ship on data you own, under a buyer-owned license, rather than data with a restricted license.

Custom collection is a poor fit if you need robot actions, teleoperation data or dense language annotation from Firsthand. The site states none of those. It is also a poor fit if you need a camera-pose or map deliverable, because the site states only the 200 Hz 6-DoF IMU. Request a custom dataset with a brief naming the stage, the tasks and the delivery format.