Guide
How to Convert RLDS to LeRobot Format
Short answer
Converting RLDS to LeRobot means re-mapping each step’s observation and action keys onto LeRobot’s parquet-plus-mp4 v2.1 layout: head_rgb becomes observation.images.head, hand and body pose get flattened into observation.state, and per-stream nanosecond timestamps get aligned to the head-camera clock. Depth does not carry over into the mp4 and must be pulled separately.
Last reviewed:
Why convert RLDS to LeRobot instead of using one format throughout?
RLDS and LeRobot are built for different parts of the pipeline. RLDS is TFDS-native and keeps every stream at its native rate, which suits large-scale training pipelines already built on tensorflow_datasets. LeRobot’s parquet-plus-mp4 layout is the smaller on-disk option and loads directly with lerobot.common.datasets, which fits teams building on LeRobot-based imitation-learning tooling. Converting lets a team keep sourcing data in one format while training in the other.
How do RLDS step and episode keys map onto LeRobot features?
- observation.head_rgb (RLDS step key)
- observation.images.head (LeRobot feature)
- No direct wrist-camera key in the base RLDS step
- observation.images.wrist_l / wrist_r (LeRobot feature)
- observation.hand_joints (RLDS step key)
- Packed into observation.state, hands first (2×21×3 flattened)
- Body pose (not a named RLDS step key on its own)
- Packed into observation.state after hands (24×3 flattened)
- observation.depth_mm (RLDS step key)
- Not embedded in LeRobot’s mp4 — pull separately from the HDF5 export
- action (RLDS step key)
- action (LeRobot feature, same name)
- Per-stream int64 nanosecond timestamps, PTP domain
- Per-frame timestamps, aligned to the head-camera clock
RLDS step/episode key to LeRobot feature mapping
These are the same field names documented on the RLDS and LeRobot format pages — the mapping does not add any keys beyond what each format already exposes.
What order does observation.state pack pose data in?
LeRobot’s observation.state packs hand joints first, then body joints, per the field mapping on the LeRobot format page. When converting from RLDS — where hand and body pose are separate step-level values — keep that ordering when flattening into the single state vector, since a loader reading observation.state will slice it assuming hands come before body.
What happens to depth when converting to LeRobot?
RLDS carries observation.depth_mm as a step key alongside RGB. LeRobot’s video-based layout does not embed depth in the mp4 stream, so a policy that consumes depth needs it pulled from the HDF5 export instead of expecting it inside the converted LeRobot dataset.
What should be checked after converting an episode?
- Confirm observation.images.head decodes to the expected resolution — RLDS keeps head_rgb at full 4K, and downscaling should happen deliberately, not silently, during conversion.
- Confirm observation.state length matches 2×21×3 hand values plus 24×3 body values in that order, per the LeRobot mapping.
- Confirm the converted dataset loads with lerobot.common.datasets.lerobot_dataset.LeRobotDataset, the same loader documented on the LeRobot format page.
- If the policy needs depth, confirm it is being read from the HDF5 export rather than assumed present in the mp4.
FAQ
How to Convert RLDS to LeRobot Format, answered.
01What is the main difference between RLDS and LeRobot that affects conversion?
RLDS stores tfrecord shards with per-stream keys at native rate; LeRobot stores parquet plus one mp4 per camera with pose flattened into a single observation.state vector. Converting means re-mapping RLDS’s separate keys into LeRobot’s consolidated feature layout.
02Does depth data survive an RLDS-to-LeRobot conversion?
Not inside the mp4. LeRobot’s video-based layout does not embed depth, so depth needs to be pulled from the HDF5 export separately if the downstream policy consumes it.
03In what order does observation.state pack hand and body pose?
Hands first (2×21×3 flattened), then body (24×3 flattened), per the LeRobot format page’s own field mapping. Keep this ordering when converting from RLDS’s separate hand and body values.
04Is there an official script for converting RLDS to LeRobot?
Firsthand does not publish a conversion script name here — the field mapping above describes how the keys correspond conceptually. Check the official RLDS and LeRobot documentation on GitHub and Hugging Face for current conversion tooling.
05Why would a team need both formats instead of picking one?
RLDS suits large-scale training pipelines already built on tensorflow_datasets and keeps streams at native rate; LeRobot is the smaller on-disk option and loads directly with LeRobot-based imitation-learning tooling. Some teams source in one and train in the other.
Sources
Where is this documented?
- RLDS (GitHub) (opens in a new tab)Official RLDS repository and step/episode structure documentation. github.com
- LeRobot (GitHub) (opens in a new tab)Official LeRobot repository, dataset loader, and v2.1 layout documentation. github.com
Check it against the sample pack.
40 episodes across 4 environments, delivered in the exact schema these guides describe.