Custom WebDataset Video Dataset for Robot Learning

WebDataset is a format in which a dataset is stored as a set of tar archives, with files that belong to one training sample grouped together by name. The WebDataset repository describes it as tar files in which files that make up a sample share a basename, with shards numbered in sequence. A custom WebDataset video dataset is egocentric video that a buyer specifies, that is captured against that written spec, and that is delivered in this layout.

This page is about ordering a collection with WebDataset as the delivery format. It is the WebDataset counterpart of the custom HDF5, custom RLDS and custom LeRobot dataset pages, for teams whose training stack streams sharded tars rather than reading a file per episode or a TFDS or LeRobot layout.

The main reason is I/O. At egocentric-video scale, reading data becomes the training bottleneck, and sharded streaming keeps accelerators fed without a random-access database. The WebDataset README makes the same point in general terms: its tar representation allows sequential I/O pipelines, and it can read from local disk or from a pipe, which lets it reach cloud object stores.

The second reason is scale beyond local disk. Firsthand ships shards that stream from signed URLs, so a team can train over more hours than it wants to stage locally. The Python quickstart shows the same idea from the command line: fh stream with a skill line and the webdataset format pipes shards straight into a trainer.

A delivery is a set of .tar shards of 512 samples each, plus the reject log for the batch and the batch data card. Each sample carries a __key__, head.jpg, wrist_l.jpg and wrist_r.jpg for the head and wrist views, depth.png for metric depth, and ann.json for the annotations.

Depth is recovered from the calibrated stereo rig rather than from a standalone depth sensor. In a shard it travels as depth.png, a 16-bit PNG that must be decoded as uint16 millimetres and not as an 8-bit image.

Which keys a given delivery holds depends on what the written skill spec ordered. The site also describes the IMU streams as shipping in WebDataset, but it does not itemise their key names or the key that carries per-stream timestamps. Those are agreed in the skill spec and confirmed on the Spec Pilot batch rather than assumed.

WebDataset is the one delivery format that differs in structure rather than in content. The site’s delivery-formats guide notes that it keys samples by frame rather than by episode, with hand-pose and action annotations delivered inline as ann.json beside the image and depth keys of that sample. The format page states that ordering is shuffled across episodes, with sequence preserved inside a sample.

Expect shards of samples rather than one file per episode, as in HDF5. How many frames one sample spans, how shards are named and file sizes are not itemised on this site, so they are set per program in the skill spec and confirmed on the Spec Pilot batch. The one stated figure is 512 samples per shard.

Whichever format you choose, the canonical field layout is the episode schema. Every delivery format is a projection of it, so switching formats later does not change what a hand-pose or wrist label contains. Stream naming stays consistent across HDF5, RLDS and WebDataset; only the container and the per-sample convention change.

The format page shows the pattern: pass the shard URLs to wds.WebDataset with shardshuffle=True, decode images, then call .to_tuple() with the keys you want, such as head.jpg, depth.png and ann.json. ann.json arrives as parsed JSON containing hand_joints and action.

Three gotchas are stated on the format page. Signed URLs expire, so regenerate them with fh stream rather than hard-coding one. depth.png is 16-bit, so decode it as uint16 millimetres. Shuffling happens across shards, so set a large enough shuffle buffer for good mixing.

They are four exports of the same QA-passed episodes, so the choice is about your pipeline, not about data availability. WebDataset suits streaming from object storage and training beyond local disk. HDF5 is the most complete and suits random access and sync verification. RLDS suits TensorFlow-based pipelines and RL-style step sequences. LeRobot suits imitation-learning pipelines on that ecosystem and is the smaller on-disk option.

The practical differences are the unit of storage and how depth and hand pose travel. WebDataset carries depth as a 16-bit depth.png per sample and hand pose inside ann.json. HDF5 keeps them as separate arrays, RLDS as per-step fields, and LeRobot folds hand pose into one state vector and leaves depth out of the mp4. If you need to verify synchronization yourself, ask for the HDF5 export alongside the shards.

The format page states that ann.json carries hand_joints, action and contact_events. hand_joints are 3D hand pose, 21 joints per hand, which the episode schema gives in metres in the head frame. The action key carries the action labelling, and the site’s annotation work elsewhere describes frame-accurate action segments with a verb and a noun.

The episode schema also lists hand_visible, instance_masks, body_joints, objects and gaze_target. Which of those appear inside a shard’s ann.json is set in the skill spec.

Every active stream is hardware-triggered onto one PTP clock domain, and the site states a cross-stream sync ceiling of 2.0 ms. That happens at capture and in QA, before any export, so it applies to the shards as it does to every other format.

On the delivery side, the glossary states that WebDataset exports preserve per-stream timestamps and metadata, and the guides state that WebDataset carries the same per-sample timestamped structure as RLDS, adapted to the format’s own convention. The exact key that holds a timestamp inside a shard is not itemised on this site.

The format that carries int64 nanosecond timestamps per stream, in the PTP domain, and lets you recompute the cross-stream residual yourself is HDF5. If synchronization verification matters to you, request it alongside the WebDataset shards. Both are built from the same episodes, so adding one adds no separate collection stage.

Every custom collection starts with a written skill spec: the task your pipeline needs covered, the setting, the object set, the capture tier, the annotation depth and the accept criteria, which are signed into it before capture begins.

For a WebDataset order the spec also names which keys go into each sample, how many frames one sample spans and that the delivery format is WebDataset. One attempt at the task is one episode, so any sample traces back to the skill and setting it came from.

Paid, consented contributors perform the real task against your spec while wearing the capture rig. The instrumented tier is a head-mounted stereo rig: first-person RGB, a metric depth stream recovered from the calibrated stereo rig, and 6-DoF IMU data at 200 Hz from head- and wrist-mounted sensors. The rig is calibrated before the session and again after it. We do not capture robot or teleoperation data, and we do not capture at a customer’s own facility.

QA happens once, against the canonical episode, before any format-specific export is produced. An episode is accepted only if RMS reprojection stays under 0.28 px and the drift is within tolerance. Otherwise it goes to the reject log, which ships with every batch. Only footage that clears the signed accept criteria is billed as a validated hour, and rejected items are recaptured at our cost.

Privacy sits in the same gate. Every episode is linked to a signed contributor release under a documented consent chain, with consent artefact IDs listed per episode in the batch data card. Faces and screens in frame are detected and irreversibly blurred at ingest, with a reviewer confirming the result before anything ships, and clips that cannot be de-identified are discarded rather than delivered.

You receive the dataset under a perpetual, irrevocable, worldwide commercial licence that is non-exclusive by default, with the buyer owning the resulting dataset and exclusivity available. The same terms apply whichever format the episodes are exported into.

The delivery is the WebDataset shards, the reject log and the data card for the batch.

Where coverage for the requested tasks and regions is already deep, the median time from signed spec to first validated batch is a median of 48 hours. Where it is thin, turnaround runs three to four weeks while recruiting expands in parallel across more regions. WebDataset adds no separate stage.

Write down the tasks, environments, objects and camera tier your model needs, name the keys and annotations you want in each sample, and specify WebDataset as the delivery format. Most orders begin with a Spec Pilot: a small, fast first batch that proves the spec and the accept criteria hold up on real footage before you scale.

On that pilot, open a shard with wds.WebDataset and check the keys, depth decoding and ann.json fields you depend on.