Custom RLDS Dataset for Robot Learning

RLDS (Reinforcement Learning Datasets) is a standard for storing episodic, step-based data for robot learning, built on TFDS. It represents each demonstration as a sequence of steps with observations and actions, which makes episodes portable across training pipelines. A custom RLDS dataset is egocentric video that is specified by a buyer, captured against that written spec, and delivered in this layout.

This page is about ordering a collection with RLDS as the delivery format. It is not the format reference: the RLDS format page holds the full key mapping, loading snippet and gotchas. It is also the RLDS counterpart of the custom LeRobot dataset page, for teams whose training stack reads TFDS rather than the LeRobot layout.

The practical reason is the pipeline. A common episodic format lets you mix datasets and reuse dataloaders instead of writing a bespoke parser for every new source, and RLDS is widely used across open robot-learning efforts. A team whose training code already imports tensorflow_datasets can load a custom collection the same way it loads everything else, and a one-off source does not need its own loader to be written and maintained.

Ordering custom data rather than relying on whatever a public dataset happened to cover means the tasks, objects and settings match what your policy is missing. RLDS also suits teams that want depth on every step: observation.depth_mm is carried per step, where the mp4-based LeRobot layout does not embed depth.

A third reason is control over the mapping. Streams are stored at native rate and the converter ships as readable source, so the field mapping can be retargeted to a loader of your own rather than accepted as a black box.

Every custom collection starts with a written skill spec: the task your pipeline needs covered, the setting, the object set, the capture tier, the annotation depth and the accept criteria. That spec is quoted for reach, timeline and price before capture begins, and the accept criteria are signed into it.

In RLDS, the skill becomes the organising unit. Shards are laid out with one skill line per directory, and each episode carries episode_metadata.skill_id and episode_metadata.site_id, so a given episode can be traced back to the skill and setting it came from. One attempt at the task is one episode; each step inside it holds the head RGB frame, the depth frame, the hand joints and the action.

Where the spec names sub-steps, the segment structure is held in the canonical episode schema as action segments with start_ns, end_ns, verb and noun. The RLDS format page does not itemise a dedicated RLDS key for them, so if your policy needs segment boundaries in the RLDS files themselves, say so in the spec and confirm the mapping on the Spec Pilot batch before you scale.

Every format on this site is a projection of one canonical episode schema with four groups: meta, streams, annotations and calibration. RLDS flattens that into the step and episode keys listed on the format page, so the table below pairs each label with where the site says it lands.

Four things are stated directly for RLDS: the head RGB frame, the metric depth frame, the hand joints and the action are step keys, and the skill, site and calibration drift are episode keys. Anything the format page does not itemise is marked as such rather than assumed, because a field you rely on should be confirmed in the Spec Pilot, not inferred from this page.

The calibration drift key is worth using. episode_metadata.calib_drift_mm lets you filter on calibration quality before training, and the per-episode reject flags that apply to a batch hold across every format, so an episode excluded for failing QA does not appear in the RLDS export either.

Paid, consented contributors perform the real task against your spec while wearing the capture rig. The instrumented tier is a head-mounted stereo rig: first-person RGB, with a metric depth stream recovered from the calibrated stereo rig rather than a standalone depth sensor, and 6-DoF IMU data at 200 Hz from head- and wrist-mounted sensors.

Every stream is hardware-triggered onto one PTP clock domain with int64 nanosecond timestamps per stream, so depth, motion and hand pose are aligned to the same video frame rather than approximated afterwards. The rig is calibrated before capture and again after the session, and the head, wrist and depth transforms are solved each time.

Capture happens in the settings this site already lists on its industry and environment pages, and collection is on demand: you do not need a standing program, and a focused brief is a normal starting point. We do not capture robot or teleoperation data, and we do not capture at a customer’s own facility.

QA happens once, against the canonical episode, before any format-specific export is produced. The RLDS shards inherit that same calibrated, synced, QA-passed data: they are a different on-disk layout of the same capture, not a separately verified pipeline.

Acceptance is measured against the signed accept criteria. An episode is accepted only if RMS reprojection stays under 0.28 px and the drift is within tolerance; otherwise it goes to the reject log, which ships with every batch. Only footage that clears the criteria is billed as a validated hour, and rejected items are recaptured at our cost.

Privacy is part of the same gate. Every episode is linked to a signed contributor release under a documented consent chain, and faces and screens in frame are detected and irreversibly blurred at ingest, with a reviewer confirming the result before anything ships. Clips that cannot be de-identified are discarded rather than delivered.

Choose the one your training stack already imports. RLDS loads with tensorflow_datasets and suits TensorFlow-based pipelines and RL-style step sequences. LeRobot loads with lerobot.common.datasets and suits imitation-learning pipelines already built on the LeRobot ecosystem. We deliver both directly, so the decision comes down to your pipeline, not data availability.

The differences that matter are depth, pose shape and footprint. RLDS carries depth per step and keeps hand_joints as its own field; LeRobot flattens hand and body pose into one observation.state vector and does not embed depth in the mp4. LeRobot is the smaller on-disk option because video stays compressed per camera, while RLDS shards store fields directly and run larger per episode.

You do not have to decide blind. Because both are exports of the same QA-passed episodes, a team can request a second format on the same Spec Pilot batch and compare loading and disk footprint on its own pipeline before scaling. If you already hold RLDS data and want LeRobot, the converting guide covers the mapping.

You receive the dataset under a perpetual, irrevocable, worldwide commercial licence that is non-exclusive by default, with the buyer owning the resulting dataset and exclusivity available. The same terms apply regardless of which format the episodes are exported into.

The delivery is the RLDS shards, the reject log for the batch, and the readable converter source. Format options for the same collection are RLDS, LeRobot, WebDataset, HDF5/zarr, or Rerun .rrd, plus raw streams with metadata, matching whatever your pipeline already loads.

Where coverage for the requested tasks and regions is already deep, the median time from signed spec to first validated batch is a median of 48 hours. Where it is thin, turnaround runs three to four weeks while recruiting expands in parallel across more regions.

Those are the same timelines as every other custom collection on this site; choosing RLDS as the format does not add a separate stage, because the export is built from the episodes that already passed QA.

Write down the tasks, environments, objects and camera tier your model needs, and specify RLDS as the delivery format. Most orders begin with a Spec Pilot: a small, fast first batch that proves the spec and the accept criteria hold up on real footage before you scale to the full order.

On that pilot, load a shard with your own pipeline and check the keys you depend on. If anything you need is not itemised on the RLDS format page, such as segment boundaries or visibility flags, name it in the spec so the mapping is agreed before volume is committed.