Delivery format
RLDS delivery format for egocentric episodes
TFDS-compatible shards, one skill line per directory, readable converter source.
By the Firsthand capture teamLast updated September 23, 2026
Short answer
RLDS (Reinforcement Learning Datasets) stores each demonstration as a sequence of steps with observations and actions, built on TFDS. Firsthand delivers TFDS-compatible tfrecord shards, one skill line per directory, with the converter shipped as readable source so you can retarget the field mapping to your own loader.
Field mapping
How Firsthand fields map into this format
- Container
- TFDS-compatible tfrecord shards
- Step keys
- observation.head_rgb, observation.depth_mm, observation.hand_joints, action, is_terminal
- Episode keys
- episode_metadata.skill_id, .site_id, .calib_drift_mm
- Shard size
- ~1.9 GB, one skill line per directory
- Timestamps
- int64 nanoseconds, PTP domain, per stream
Loading
Loading it
import tensorflow_datasets as tfds
ds = tfds.builder_from_directory("data/kitchen_knife_dice").as_dataset(split="all")
for episode in ds:
for step in episode["steps"]:
rgb = step["observation"]["head_rgb"] # [2160, 3840, 3]
hands = step["observation"]["hand_joints"] # [2, 21, 3] metres
action = step["action"]Gotchas
What to watch for
- Streams are stored at native rate; resample in your pipeline, not by re-encoding shards.
- head_rgb is full 4K — decode lazily or downscale in the map fn to keep the input pipeline fed.
- episode_metadata.calib_drift_mm lets you filter on calibration quality before training.
Get a RLDS sample.
40 episodes across 4 environments, delivered in the format your stack already reads.