Comparison
RLDS vs LeRobot
Short answer
RLDS stores each episode as TFDS-compatible tfrecord shards with full per-stream fields, built for TensorFlow-based pipelines. LeRobot stores episodes as parquet plus one mp4 per camera in the Hugging Face datasets layout, packing hand and body pose into a single observation.state vector. Firsthand delivers both directly, so the choice comes down to your training stack, not data availability.
Spec comparison
RLDS vs LeRobot, field by field.
Evaluation criteria
Eight things to check on any vendor.
- Training stack
- RLDS is built on TFDS and loads with tensorflow_datasets. LeRobot loads with lerobot.common.datasets and is the layout the LeRobot ecosystem expects natively. Pick the one your training code already imports.
- Depth access
- RLDS carries observation.depth_mm as its own per-step field. LeRobot’s mp4-based layout does not embed depth, so a policy that consumes depth needs the HDF5 export alongside the LeRobot files.
- Pose field shape
- RLDS keeps hand_joints as its own field per step. LeRobot flattens 2×21×3 hand joints and then 24×3 body joints into one observation.state vector — keep that ordering in mind when slicing it.
- On-disk footprint
- LeRobot’s parquet + mp4 layout is the smaller on-disk option since video is compressed per camera; RLDS shards store fields directly and run larger per episode.
Red flags
Walk away when you see these.
- Assuming either format includes depth by default — LeRobot does not embed it in the mp4; pull it from the HDF5 export if your policy needs it.
- Decoding LeRobot mp4 per item instead of using the dataset’s built-in video backend, which becomes the bottleneck.
- Losing the hand-then-body ordering when slicing LeRobot’s flattened observation.state vector.
Scoring template
Score candidates against your skill spec.
Copy this table, weight each criterion for your use case, and score each vendor on the same free or paid sample.
FAQ
Common questions.
What is the difference between RLDS and LeRobot?
RLDS stores each episode as TFDS-compatible tfrecord shards with observation and action fields kept separately, built for TensorFlow pipelines. LeRobot stores episodes as parquet plus one mp4 per camera in the Hugging Face datasets layout, packing hand and body pose into a single observation.state vector.
Which format should I use for imitation learning?
If your pipeline is already built on the LeRobot ecosystem, use LeRobot — it is the native layout and loads with lerobot.common.datasets. If your pipeline is TensorFlow-based or expects RL-style step sequences, RLDS is the better fit.
Does RLDS or LeRobot include depth data?
RLDS carries depth directly as observation.depth_mm on every step. LeRobot does not embed depth in its mp4 files — pull it from the accompanying HDF5 export if your policy consumes depth.
Does Firsthand deliver in both formats?
Yes. Firsthand exports both the RLDS tfrecord layout and the LeRobot v2.1 layout directly, alongside WebDataset, HDF5, zarr and Rerun .rrd, so you can pick whichever matches your training stack without a separate conversion step.
Can I convert between RLDS and LeRobot myself?
Both formats project from the same underlying episode schema, so a conversion is possible, but it is simpler to request the format your pipeline already expects directly — Firsthand ships the converter as readable source for RLDS specifically so you can retarget the field mapping if needed. See the RLDS-to-LeRobot conversion guide for the field-by-field mapping.
Last reviewed: 2026-09-23
Judge it on your own skill spec.
Get the free 40-episode sample pack, or send us your target skills and hours for a scoped quote.