What Format Should Robot Pretraining Data Be Delivered In?

The right answer depends on what stage of the pipeline the data feeds and how it will be loaded, not on which format is objectively "best." A pretraining corpus is large enough that the wrong container adds real friction: a format that does not stream well forces you to stage everything locally first, and a format that does not match your training library forces a conversion step before you can start a run. The starting question is not "which format is best" but "which format does my training code already expect, and does the corpus need to stream or fit on local disk."

Every Firsthand delivery format is a projection of the same underlying episode schema — meta, streams, annotations and calibration groups with explicit dtypes, shapes and metric units, documented on the episode schema page. That means the choice of RLDS, LeRobot, WebDataset, HDF5/zarr or Rerun .rrd is a packaging decision, not a data-completeness one: no format is missing a field the others have, except where a format is architecturally unable to embed something (LeRobot’s mp4 layout does not carry depth inline, for example).

All six are documented on their own format pages with a field mapping, a loading snippet and known gotchas — see the formats hub.

For a pretraining-scale corpus specifically, the deciding factor is usually how the data will be fed into training, not a feature difference between the formats. RLDS keeps every stream at native rate in TFDS-compatible shards, which suits large-scale pipelines already built on tensorflow_datasets and is the format that keeps depth (observation.depth_mm) directly on each step. LeRobot is the smallest on-disk option because video is compressed per camera into mp4, and it loads natively into LeRobot-based imitation-learning tooling — but its mp4-based layout does not embed depth, so a policy that needs depth pulls it from the accompanying HDF5 export. WebDataset is built for streaming: shards are .tar archives read directly from signed URLs, which matters when a pretraining corpus is larger than a team wants to stage on local disk before a run starts.

This table restates the same facts already documented on the RLDS, LeRobot and WebDataset format pages and on the RLDS vs LeRobot comparison page — nothing here is a new claim.

Sharding decisions are already made by the format, not by the buyer, which simplifies the request: RLDS ships in ~1.9 GB shards with one skill line per directory, WebDataset ships in .tar shards of 512 samples with shuffling applied across shards (sequence preserved inside a sample), and LeRobot’s parquet-plus-mp4 layout is organized per the v2.1 dataset spec. For a pretraining corpus spanning many skill lines, the practical question is whether a training loop needs random access across the whole corpus (favoring HDF5/zarr, where one group per episode with time-major chunking supports it) or sequential, streamed reads (favoring WebDataset’s signed-URL model or RLDS’s TFDS-native sharding).

• WebDataset shards stream directly from signed URLs and work with wds.WebDataset(url) without staging locally first.
• RLDS shards are read with tfds.builder_from_directory, one skill line per directory, at roughly 1.9 GB per shard.
• HDF5/zarr chunks are time-major and 30 frames per chunk — read whole chunks rather than single frames to avoid decompression overhead across a large corpus.
• Streams inside any format are kept at native rate and resampled by the loader, never re-encoded into the delivered shards — this holds across RLDS, LeRobot, WebDataset and HDF5/zarr alike.
• WebDataset’s signed URLs expire, so regenerating them through the appropriate streaming tooling rather than hard-coding a URL matters more at pretraining scale, where a training run may span many hours of continuous shard reads.

Regardless of delivery format, an episode follows the same four-group episode schema: meta.json (episode_id, skill_id, site_id, participant_id, duration_ns, license, reject_flags), a streams/ group (head_rgb, wrist_l/r, depth_mm, depth_conf, imu_head, imu_wrist_l/r, and per-stream *_timestamp_ns), an annotations/ group (hand_joints, hand_visible, body_joints, contact_events, instance_masks, objects, action_segments, gaze_target), and a calibration/ group (intrinsics, extrinsics as SE3, rms_reproj_px). Units are metric throughout — depth in millimetres, hand joints in metres — and every format packages a projection of these same four groups differently, not a different set of fields.

Depth, IMU and hand pose are the fields most likely to differ in how (or whether) a given format embeds them, so it is worth mapping each explicitly rather than assuming parity across formats. RLDS keeps depth as observation.depth_mm and hand pose as observation.hand_joints, both directly on the step. LeRobot flattens hand joints (2×21×3) and then body joints (24×3) into a single observation.state vector, and does not embed depth in its mp4 files at all — that has to come from the HDF5 export. WebDataset carries depth as depth.png (16-bit, decode as uint16 millimetres, not 8-bit) and hand pose inside ann.json alongside action and contact_events. HDF5/zarr stores everything under streams/ and annotations/ exactly as the episode schema defines it, which is why it is the format Firsthand’s own QA tooling reads.

Yes, because every format is a projection of the same episode schema, but converting after the fact means re-mapping keys rather than gaining new fields. The RLDS-to-LeRobot conversion guide walks through exactly this: observation.head_rgb becomes observation.images.head, hand and body pose get flattened into observation.state in a fixed hands-then-body order, and depth has to be pulled from the HDF5 export since it is not embedded in LeRobot’s mp4 layout. It is usually simpler to request the format your pipeline already expects at delivery time than to convert afterward, but Firsthand ships the RLDS converter as readable source specifically so the field mapping can be retargeted if your loader needs something different.

The delivery format is one line in a collection spec, alongside the skill list, streams and label set — it does not change the price or the collection process, since every format is generated from the same captured episodes. Name the format your training stack already loads (RLDS for TFDS pipelines, LeRobot for the smallest footprint, WebDataset for shard-streamed training, HDF5/zarr if you want to verify sync yourself) when requesting a custom pretraining dataset, and Firsthand exports it directly rather than requiring a separate conversion step after delivery.