Custom HDF5 Robot Dataset

HDF5 is a hierarchical file format that organises data as groups and datasets, a little like folders and arrays, and it is read in Python with h5py. A custom HDF5 robot dataset is egocentric video that a buyer specifies, that is captured against that written spec, and that is delivered in this layout: one group per episode, with the streams, annotations and calibration for that episode inside it.

This page is about ordering a collection with HDF5 as the delivery format. It is the HDF5 counterpart of the custom RLDS dataset and custom LeRobot dataset pages, for teams whose training or evaluation code reads arrays from a file rather than a TFDS or LeRobot layout.

The first reason is completeness. HDF5 is the format the Firsthand QA tooling reads, so it is the most complete of the delivery formats: nothing is folded into a state vector or left in a separate export. If a team wants to see every stream and every annotation exactly as QA saw it, this is the delivery to ask for.

The second is access pattern. Episodes are stored as one group each, chunked time-major, which suits random access into long episodes. A training loop that samples windows, or an analysis script that inspects one stretch of an episode, reads just the chunks it needs.

The third is verification. HDF5 carries the per-stream nanosecond timestamps, so a team can verify synchronization itself.

The streams/ group carries the head RGB stream, the two wrist RGB streams, metric depth with a confidence channel, and the IMU streams, each with its own timestamps. Which streams a given delivery actually holds depends on which the written spec ordered: a perception model may need depth more than IMU, while a manipulation policy may lean on wrist motion.

Depth is recovered from the calibrated stereo rig rather than from a standalone depth sensor, and it is registered to the head-camera frame using the solved calibration extrinsics. The three IMUs are 6-DoF sensors, one on the head and one on each wrist, so each is its own array and can be analysed alone or together.

Each episode is one .h5 file (or one .zarr group) named for the episode, such as EP_0117_KITCHEN_A.h5. Inside it are three groups that mirror the canonical episode schema: streams/, annotations/ and calibration/. The episode’s identity and licence live in meta.json under the same schema: episode_id, skill_id, site_id, participant_id, duration_ns, license and reject_flags.

Units are metric throughout. Depth is in millimetres and hand joints are in metres, in the head frame. T is the number of head-camera frames and N is the number of IMU samples, and they differ because the streams run at different rates. Streams are stored at native rate and resampled by your loader, never re-encoded in the delivery.

The calibration/ group carries intrinsics, extrinsics as SE3 transforms, and rms_reproj_px, along with both the opening and the closing solve, so you can check drift across a session. The head-camera frame is the identity for all extrinsics. For object storage, zarr is the better choice; for a single local file, HDF5.

Chunk shape and compression are stated on the format page as time-major, 30-frame chunks with blosc-zstd level 5, so read whole chunks rather than single frames to avoid decompression overhead. Per-stream chunk shapes beyond that, file sizes and shard sizes are not itemised on this site, and are set per program in the skill spec.

The annotations/ group holds the labels. The ones most orders ask for are hand_joints, a float32 array of shape [T, 2, 21, 3] in metres, so 21 joints per hand in 3D; hand_visible, the per-joint visibility flag; contact_events; and action_segments, a list of start_ns, end_ns, verb and noun entries that mark sub-steps of a skill.

The episode schema also lists instance_masks (uint16, [T, 480, 848]), body_joints, objects and gaze_target in the same group. Whether a given delivery includes them is a spec decision, so name the ones your policy needs in the skill spec and confirm them on the Spec Pilot batch rather than assuming them.

They are three exports of the same QA-passed episodes, so the choice is about your pipeline, not about data availability. HDF5 is the most complete and suits random access and sync verification. RLDS suits TensorFlow-based pipelines and RL-style step sequences, with depth carried per step. LeRobot suits imitation-learning pipelines on that ecosystem and is the smallest on disk.

The practical differences are depth, pose shape and footprint. HDF5 keeps depth, IMU and hand joints as separate arrays; RLDS keeps hand_joints as its own field per step; LeRobot flattens 2×21×3 hand joints, then 24×3 body joints, into one observation.state vector and leaves depth out of the mp4.

Every active stream is hardware-triggered onto one PTP clock domain, and each stream carries int64 nanosecond timestamps in that domain. The site states a cross-stream sync ceiling of 2.0 ms, hardware-triggered. The IMUs are aligned during calibration with a static hold plus a figure-eight excitation, and the stereo rig that produces depth is validated in the calibration pass.

Because the IMUs sample at 200 Hz and the cameras at 30 or 60 Hz, an episode holds more IMU samples than video frames for the same span of time. Consuming them together means resampling or windowing the IMU data to the frame you need, rather than assuming one sample per frame.

The HDF5 export carries the per-stream timestamps, so you can recompute the residual yourself. The head_rgb dataset exposes them as a timestamp_ns attribute, and the sync-error guide shows how the residual is measured. The Rerun recording plots the same residual against the ceiling if you would rather look first.

Every custom collection starts with a written skill spec: the task your pipeline needs covered, the setting, the object set, the capture tier, the annotation depth and the accept criteria. That spec is quoted for reach, timeline and price before capture begins, and the accept criteria are signed into it.

For an HDF5 order the spec also names which streams and annotations go into each episode group and that the delivery format is HDF5, zarr or both. One attempt at the task is one episode and one group, tagged with skill_id and site_id in meta.json, so any episode traces back to the skill and setting it came from.

Paid, consented contributors perform the real task against your spec while wearing the capture rig. The instrumented tier is a head-mounted stereo rig: first-person RGB, a metric depth stream recovered from the calibrated stereo rig, and 6-DoF IMU data at 200 Hz from head- and wrist-mounted sensors. The rig is calibrated before the session and again after it. We do not capture robot or teleoperation data, and we do not capture at a customer’s own facility.

QA happens once, against the canonical episode, before any format-specific export is produced, so the HDF5 files inherit the same calibrated, synced, QA-passed data as every other format. An episode is accepted only if RMS reprojection stays under 0.28 px and the drift is within tolerance. Otherwise it goes to the reject log, which ships with every batch. Only footage that clears the signed accept criteria is billed as a validated hour, and rejected items are recaptured at our cost.

Privacy sits in the same gate. Every episode is linked to a signed contributor release under a documented consent chain. Faces and screens in frame are detected and irreversibly blurred at ingest, with a reviewer confirming the result before anything ships, and clips that cannot be de-identified are discarded rather than delivered.

You receive the dataset under a perpetual, irrevocable, worldwide commercial licence that is non-exclusive by default, with the buyer owning the resulting dataset and exclusivity available. The same terms apply whichever format the episodes are exported into.

The delivery is the HDF5 (or zarr) episodes and the reject log for the batch.

Where coverage for the requested tasks and regions is already deep, the median time from signed spec to first validated batch is a median of 48 hours. Where it is thin, turnaround runs three to four weeks while recruiting expands in parallel across more regions. Choosing HDF5 adds no separate stage, because the export is built from episodes that already passed QA.

Write down the tasks, environments, objects and camera tier your model needs, name the streams and annotations you want in each episode group, and specify HDF5 or zarr as the delivery format. Most orders begin with a Spec Pilot: a small, fast first batch that proves the spec and the accept criteria hold up on real footage before you scale.

On that pilot, open an episode with h5py and check the stream paths and shapes you depend on. Anything not itemised on the format pages is agreed in the spec before volume is committed.