VLA Fine-Tuning Data

VLA fine-tuning data is the task-specific data used to adapt a vision-language-action model that is already pretrained. A VLA model takes a camera view and a language instruction and outputs an action. Fine-tuning, which some papers also call post-training, starts from the pretrained weights and updates them on a smaller, narrower dataset so the model handles your tasks, objects and setting.

This page is for teams that already hold a pretrained VLA and need that narrower dataset. People search for it as VLA fine-tuning data, a vision-language-action fine-tuning dataset, a custom VLA dataset, or data to fine-tune a model such as OpenVLA or π0. Firsthand collects custom egocentric data on demand to a written spec, and that is the service described here.

This page does not claim that Firsthand data trained, fine-tuned or evaluated any named model, and it does not claim any affiliation with the authors of the models it cites. Third-party statements come only from the official papers and repository linked in Sources. Everything about Firsthand comes from other pages of this site, and where the site states nothing, this page says Not stated.

The two stages ask different things of a dataset. Pretraining data is planned for breadth: many scenes, tasks and objects, so the model builds a general prior. Fine-tuning data is planned for precision: how closely it matches the task the model has to do. The VLA pretraining data page sets this out, and the first table below carries the comparison.

Three pages on this site touch fine-tuning, and they answer different questions. The VLA pretraining data page covers the stage before this one. The fine-tuning data for robot policies page covers post-training for any robot policy, including the written skill spec, the Spec Pilot and failure and recovery coverage. This page narrows to the vision-language-action model: its inputs, what the official papers say about adapting it, and how to scope a collection for it. If your model is not a VLA, start with the robot policy page.

Mid-training sits between pretraining and fine-tuning. The EgoScale abstract describes large-scale human pretraining followed by lightweight aligned human-robot mid-training, so a team following that recipe may need the mid-training data page as well as this one.

Each source below was read at the link given in Sources, and only what it states is repeated here. The RT-2 abstract describes co-fine-tuning vision-language models on both robotic trajectory data and Internet-scale vision-language tasks, with actions expressed as text tokens. The OpenVLA abstract says that, rather than training new behaviours from scratch, such models can be fine-tuned to obtain policies for visuomotor control, and that OpenVLA can be fine-tuned on consumer GPUs through low-rank adaptation.

The OpenVLA repository describes three fine-tuning modes: full, partial and LoRA. It states that its fine-tuning script is compatible with datasets in RLDS format, and that for a custom dataset outside its supported mixture you can convert to RLDS or write your own PyTorch dataset wrapper. It also states that a new dataset has to be registered with its data loader through a dataset config and a transform function. It recommends full fine-tuning only if LoRA is insufficient, for example when the fine-tuning distribution varies drastically from the pretraining distribution.

The π0 abstract says the model is evaluated on its ability to acquire new skills via fine-tuning, after training on data from multiple dexterous robot platforms. The Octo abstract says generalist policies may be finetuned with only a little in-domain data, and the paper reports finetuning setups that each use around 100 in-domain demonstrations. The GR00T N1 paper describes a post-training phase that fine-tunes the pretrained model on datasets corresponding to each single embodiment.

Two points from these sources shape a data spec. First, in a qualitative example the GR00T N1 paper notes that all post-training data involved only the right hand with no inter-hand transfer, and that the post-trained policy lost a handover behaviour the pretrained checkpoint showed. A fine-tuning set narrows what the model does, so the spec has to list every behaviour you want to keep. Second, the same paper notes that for the open human egocentric video it used there were no actions it could use directly, and it describes learning latent actions to label that video. Human video and robot actions are different things, and how one maps to the other is a decision for your pipeline.

Each Firsthand episode follows the episode schema: ten streams synced under 2 ms, with head and wrist RGB, depth, 3D hand pose with per-joint visibility, contact events on grasp and release, and frame-accurate action segments. The site also states 200 Hz 6-DoF IMUs on the Firsthand capture rig. Whether a given collection includes them is set in the spec.

The one language field the site states is a task instruction paired to each episode, on the same synchronized clock as the video and action stream. That is the whole of the language data this page describes: one task instruction per episode. The site does not describe narration, captions, dense language annotation, multi-turn dialogue, subtask language or reasoning traces as deliverables, so none is claimed.

The action in these episodes is human action: 3D hand pose, contact events and action segments. The official sources describe models that output robot actions, such as the 7-DoF action in the OpenVLA repository example. The site does not state that Firsthand delivers robot action logs, teleoperation data, robot embodiments or action spaces, so those are Not stated here. If your fine-tuning stack needs robot actions, confirm in the spec how the human signals are to be mapped, and who does the mapping, before you plan around the data.

Object labels, success labels and quality labels use whatever terms your spec sets, because the site states no fixed vocabulary for them. This page makes no head or camera pose, mapping or SLAM claim as a deliverable.

Fine-tuning is scoped to one deployment, so the task list comes from you. The environments and task catalog on this site lists commercial environments with tasks, a skill rating and a status for each, and a written spec can draw on those rows or name the task you need. Which tasks a given buyer can reach, and in which environments, is quoted against the spec before capture begins.

The robot policy fine-tuning page states that capture is head-mounted or wrist-mounted and matched to the target task, with RGB, depth, IMU and calibrated intrinsics. It also states that a fine-tuning spec can name the failure modes and recovery behaviours a policy currently struggles with, so a batch is built around those and not only the clean version of the task. The failure and recovery demonstration data page covers that in more detail.

Robot embodiments, action spaces, task durations, the number of sites and the number of episodes are Not stated, because the site publishes no figure for them. They are set in the written spec and the quote.

A useful scoping habit, drawn from the GR00T N1 qualitative example above, is to write down the behaviours the pretrained model already does well and that you need to keep, then ask for fine-tuning data that includes them alongside the new task.

Delivery formats are RLDS, LeRobot and WebDataset, plus HDF5, zarr and Rerun .rrd. Each is a projection of the episode schema, so the same episode can be read from whichever format your training code expects. The fourth table below sets each format against what the site states and what the sources say.

RLDS matters most for one named stack. The OpenVLA repository states that it natively supports datasets in RLDS format and recommends converting a custom dataset to RLDS for most users. The site states that its RLDS delivery ships the converter as readable source, so you can retarget the field mapping to your own loader. Whether the Firsthand RLDS field names match what that repository expects is Not stated, and this page does not claim that any Firsthand delivery has been run through it. Expect a short mapping step.

For the other formats, the sources read here state nothing about fine-tuning compatibility, so none is claimed. Choose the format your own training code reads and say so in the spec.

Every episode links to a signed contributor release through a documented consent chain, and the consent record ships with the delivery. The site states that faces, plates and screens are detected and irreversibly blurred at ingest and that a reviewer confirms the result. The anonymization record is delivered with the data.

That is the whole of the privacy wording this page relies on. The site states no rule for data stored on a device, and none is claimed here. Every collection ships under a perpetual, buyer-owned commercial license.

The site publishes two figures. Where coverage is already deep, the first validated batch lands in a median of 48 hours from signed spec. Where coverage is thin, it typically takes three to four weeks, because vetting operators there is the bottleneck, and the quote says which applies. Neither is a delivery date for your dataset, and the site states no batch size or episode count.

Because fine-tuning batches are small, the usual route is a Spec Pilot first: a small, fast batch that proves the spec and accept criteria on real footage before you scale. After each training run, on-demand top-ups run against the same spec and accept criteria.

Billing is per validated hour, which means footage that passed sync and QA against your spec. A reject log ships with every batch, and rejected episodes are recaptured at our cost.

Write down the model you are fine-tuning and the stage it is at, the tasks and objects it must handle, and the behaviours it already does that you must not lose. We turn that into a written spec covering tasks, environments, capture tier, annotation depth and accept criteria, then quote reach, timeline and price before any capture begins.

Things worth settling in the spec are the wording of the single task instruction for each episode, which human signals your pipeline will use, the format you want, how many episodes to hold back for evaluation, and the success label if you want one. The site states none of these, which is why the spec is where they are written down.

Billing, the reject log, recapture at our cost and the perpetual buyer-owned license apply to a pilot as they do to a full order.