VLA Dataset Format: What Does a Training Episode Contain?

Across the sources read for this guide, a VLA training episode is one demonstration trajectory: a sequence of timesteps, each pairing an observation with the action taken at that moment, plus a language instruction describing the task. OpenVLA’s own description states the setup directly: "an input observation image and a natural language task instruction are mapped to a string of predicted robot actions" (https://arxiv.org/abs/2406.09246). The Octo paper describes "a transformer architecture that maps arbitrary input tokens (created from observations and tasks) to output tokens (then decoded into actions)" (https://arxiv.org/abs/2405.12213). Octo is described by its own paper as a generalist robot policy, so this guide does not label it a VLA.

What differs between papers is how much structure sits inside each of those three parts, and whether the action is a discrete token or a continuous vector. The sections below take observations, actions and language in turn, each attributed to the source that states it, then show how Firsthand’s own episode fields, as named on this site, map onto that structure.

OpenVLA’s GitHub repository documents its data loading as natively supporting "arbitrary datasets in RLDS format, including arbitrary mixtures of data from the Open X-Embodiment Dataset" (https://github.com/openvla/openvla), and its fine-tuning instructions say to add "a new configuration specifying your fine-tuning dataset’s observation and action spaces" (https://github.com/openvla/openvla). Not stated in the sources read: the full list of observation keys the OpenVLA loader requires.

Octo’s paper describes its training mixture as heterogeneous "in the sensors (e.g., including or not including wrist cameras)" and says it will "zero-pad any missing camera channels" (https://arxiv.org/abs/2405.12213), so wrist-camera observations are optional per source dataset rather than a fixed requirement. The paper reports that "only 27% of the data contains wrist camera information." The π0 paper describes a richer observation for its own data: "multiple RGB images, a language command, and the robot’s proprioceptive state," written as o_t = [I¹_t, ..., Iⁿ_t, ℓ_t, q_t] where q_t is "a vector of joint angles" (https://arxiv.org/abs/2410.24164). Not stated in the sources read: a single universal observation schema that every VLA dataset format enforces; each paper describes what its own training mixture happened to include.

RT-2 represents each action dimension as a discrete token. Its paper states the recipe plainly: "The continuous dimensions ... are discretized into 256 bins uniformly," producing an action string such as "1 128 91 241 5 101 127" that is inserted into the model’s text output alongside language tokens (https://arxiv.org/abs/2307.15818). OpenVLA follows the same approach with one refinement: it discretizes "each dimension of the robot actions separately into one of 256 bins," but sets bin edges from "the 1st and 99th quantile of the actions in the training data" rather than the min and max, "to ignore outlier actions" (https://arxiv.org/abs/2406.09246).

π0 does the opposite: it states explicitly that prior VLAs "employ autoregressive discretization to represent actions," and that its own model instead "produce[s] actions via flow matching," a continuous diffusion-style method, specifically to "handle high-frequency action chunks (up to 50 Hz) and highly dexterous tasks" that the paper says "pose a major challenge for prior autoregressive VLAs" (https://arxiv.org/abs/2410.24164). Octo also uses a continuous head: the π0 paper states that "While Octo is not a VLA, it does use a diffusion process to generate actions" (https://arxiv.org/abs/2410.24164), and Octo’s own paper describes output tokens "then decoded into actions" (https://arxiv.org/abs/2405.12213).

RT-2’s paper states that each demonstration in its training data "is annotated with a natural language instruction that describes the task performed, consisting of a verb describing the skill ... and one or more nouns describing the objects manipulated" (https://arxiv.org/abs/2307.15818). Octo "can be guided via either language commands or goal images," and describes its mixture as heterogeneous in labels "(e.g., including or not including language instructions)" (https://arxiv.org/abs/2405.12213) — so language is a per-dataset optional field in Octo’s own mixture, not a universal requirement. The paper reports that "only 56% of the pretraining data contains language annotations."

None of the sources read for this guide describe a VLA dataset format that requires more than one instruction string per episode or per labeled segment. π0’s paper does describe "segment annotations (fine-grained labels for sub-trajectories, typically about 2 seconds in length)" as part of its own pre-training mixture (https://arxiv.org/abs/2410.24164), which is a finer-grained language structure than a single whole-episode instruction, but it is specific to π0’s own data and not stated as a format requirement shared across RLDS, LeRobot or OpenVLA.

Firsthand’s own language field for delivered episodes is one task instruction per episode. The sources above describe richer language structures some VLA papers use in their own training data — that is a fact about those papers’ datasets, not a claim about what Firsthand captures.

OpenVLA’s repository names RLDS as its native training format and documents the fallback path for a dataset that is not already in Open X-Embodiment: "convert the dataset to the RLDS format which is compatible with our fine-tuning script" (https://github.com/openvla/openvla). GR00T N1’s paper names a different format as its foundation: "Our training corpora build upon the LeRobot dataset format (Cadene et al., 2024)," developed by Hugging Face, and it extends the format so that state and action vectors are split into "semantically meaningful fields" (https://arxiv.org/abs/2503.14734). Octo trains on "a mixture of 25 datasets from the Open X-Embodiment Dataset" (https://arxiv.org/abs/2405.12213); which storage format Octo’s loader reads is Not stated in the sources read.

Firsthand delivers the same underlying episode in five containers: RLDS, LeRobot v2.1, WebDataset, HDF5/zarr, and a Rerun .rrd recording for inspection rather than training. The table below lists each Firsthand field as this site names it, alongside the VLA episode role it fills and the format page where it is documented.

The episode schema page documents the full field list this table draws from (/formats/episode-schema), and the RLDS and LeRobot format pages document the step and feature keys exactly as delivered (/formats/rlds, /formats/lerobot).

Firsthand’s delivered episode carries continuous, not pre-tokenized, fields for the observation and action side of a VLA episode: 3D hand pose (annotations/hand_joints, a [T, 2, 21, 3] array in metres) with a per-joint visibility flag (hand_visible), wrist trajectories, timestamped action segments (start_ns, end_ns, verb, noun), contact events, 200 Hz IMU, and calibrated stereo depth. One task instruction per episode is the language field. None of these fields arrive as RT-2 or OpenVLA-style discrete action-bin tokens; a buyer’s own pipeline performs any tokenization, chunking, or retargeting to a specific robot’s action space after delivery.

Every format described above was designed around robot demonstration data: a robot’s own proprioceptive state and the action it executed. Firsthand captures first-person human video, not robot or teleoperation data, so an episode has no robot joint state, no gripper action, and no robot-executed action vector to drop directly into the action field a format like RLDS or LeRobot expects from a robot dataset. Bridging that gap, deciding how a measured human hand motion maps to a given robot’s joints and action space, is a step the buyer’s own pipeline performs; it is not part of what any format conversion alone does.

This is a narrower and more literal point than the embodiment-gap discussion in the sources themselves. GR00T N1’s paper describes training on human video that has no ground-truth actions at all, and compensates with a learned latent-action codebook: "For human egocentric videos ... we do not have any actions that we can directly use to train GR00T N1. For these data, we instead generate latent actions by training a VQ-VAE model" (https://arxiv.org/abs/2503.14734). EgoScale’s paper takes a different approach for its own human data: it supervises the model with human manipulation behaviours "represented as relative wrist motion and retargeted high-DoF hand joint actions," and after pretraining introduces "a small amount of aligned human-robot mid-training data" (https://arxiv.org/abs/2602.16710). Both are solutions a model-training pipeline applies to action-less human video; neither claims that a delivered human-video format on its own already contains a robot action field.

• Name the format your training stack already reads, RLDS, LeRobot, WebDataset, HDF5/zarr, or a Rerun .rrd for inspection first.
• Confirm which fields your action head needs as continuous inputs, since Firsthand delivers hand pose, wrist trajectories, action segments and contact events as continuous fields, not pre-discretized tokens.
• Keep the language request exact: one task instruction per episode is the field this site states for delivered capture.
• Decide where embodiment bridging happens. Firsthand captures first-person human video, not robot or teleoperation data, so retargeting to a specific robot’s action space is a step your own pipeline performs after delivery.
• Request a sample in your target format and load it with your existing data loader before committing to a full custom collection, on demand and fast to turn around.

That written spec, naming the format and the fields, is what a custom collection batch starts from.