Custom Egocentric Dataset Delivered in Rerun (.rrd) Format

Rerun is a visualization tool for multi-stream data, and its recordings are stored in .rrd files; the maintainers describe a recording, in its simplest form, as an individual .rrd file containing Rerun data. A custom egocentric dataset in Rerun format is first-person video that a buyer specifies, that is captured against that written spec, and that is delivered as one such recording per episode.

This page is about ordering a collection with Rerun recordings as the delivery. It is the Rerun counterpart of the custom HDF5, custom RLDS, custom LeRobot and custom WebDataset pages. Those four deliver data for a loader to read; this one delivers data for a person to look at, so a team that also trains on the data names a training format in the spec as well.

Rerun is a third-party tool documented by its own maintainers. This site states no partnership with it, and the facts about the .rrd file, the viewer, entity paths and timelines below come from Rerun’s own documentation and repository, linked in the sources.

The format page gives the reason directly: a Rerun recording puts every stream on one timeline, and it is how Firsthand recommends a new team first meets the dataset, before writing a loader, so you can see the alignment with your own eyes. Multimodal egocentric data is hard to judge from a file listing. Head video, hand pose, a point cloud and a sync-error plot are separate streams, and the quickest way to judge whether they agree is to scrub through them side by side.

The second reason is review. A recording is a single file you open with the viewer, so a researcher, an annotation lead and a procurement reviewer can look at the same episode without writing code. The third is acceptance: you can check a Spec Pilot batch visually against the skill spec before committing volume.

What it is not: the format page says plainly that this is for inspection, not training, and to pull RLDS, LeRobot or WebDataset for a training loop. A Rerun-only order suits a team that wants to evaluate the data or review captures. A team training a policy should name a training format as well.

The format page lists four entities in an episode recording: world/head_cam, world/hands, world/points and plots/sync_error. Its loading note describes them as the camera, the hand skeleton, the point cloud and the sync-error plot, all staying aligned as you scrub to any moment. A separate guide adds that depth is surfaced visually on the shared timeline.

The episode schema behind every delivery lists more than that: head RGB at 60 Hz, two wrist RGB streams at 60 Hz, metric depth at 30 Hz, and head and wrist IMUs at 200 Hz, each with int64 nanosecond timestamps. The site does not state that the wrist cameras or the IMUs appear as entities in the .rrd. If your review depends on seeing one of them in the viewer, name it in the skill spec and confirm it on the Spec Pilot batch rather than assuming it. The first table below sets out what is stated and what is not.

Depth is recovered from the calibrated stereo rig rather than a standalone depth sensor. The point cloud entity is listed on the format page, but the site does not state how it is built or at what density, so ask for those details in the spec if you rely on it.

Only hand pose is stated as visible in the recording: the world/hands entity carries the hand skeleton. In the episode schema that label is annotations/hand_joints, a float32 array of shape [T, 2, 21, 3] in metres in the head frame, so 21 joints per hand in 3D.

Action segments (a list of start_ns, end_ns, verb and noun) and contact events are listed in the annotations group of the episode schema, and the WebDataset page states that its ann.json carries hand pose, action and contact events. The site does not state that either appears as a track or overlay in the .rrd. Instance masks, body joints, objects and gaze target are likewise schema fields that are not stated as Rerun entities.

In practice, if you want to review a label visually, say so in the spec and confirm it on the Spec Pilot recording. If you need the label as data, take it from the HDF5 or WebDataset export of the same episodes. The second table below goes label by label, with “Not stated” wherever the site is silent.

Two Rerun concepts matter for reading a delivery. Rerun arranges entity paths in a hierarchy with the / character as the separator, which matches the way the format page groups its entities: world/ for the scene and plots/ for the chart. And each piece of logged data is associated with one or more timelines. Rerun supports sequence indices as well as timestamps and durations in nanoseconds, and you choose which timeline to organise the view along in the viewer.

The site states that all streams sit on one shared nanosecond timeline, which is consistent with the per-stream int64 nanosecond timestamps in the PTP clock domain that every Firsthand delivery carries. It does not say which other timelines, if any, a recording contains, or what the timeline is named. Both are easy to read off the Spec Pilot recording.

Rerun’s documentation also notes that RRD files saved with Rerun 0.23 or later can be opened with a newer Rerun version. The site states no Rerun version for its recordings, so if your viewer is pinned, put the version you run in the skill spec and open the Spec Pilot file with it first. The third table below pairs each Rerun concept with how this delivery uses it.

The format page gives the commands: pip install rerun-sdk, then rerun EP_0117_KITCHEN_A.rrd to open the recording in the Rerun viewer. The filename there is the page’s example, not a stated naming rule for every delivery. Rerun’s own documentation says the same thing in general terms: each SDK can save logged data to disk, and the resulting file is viewed with rerun path/to/recording.rrd.

Once it is open, every stream shares one timeline, so you scrub to any moment and the camera, hand skeleton, point cloud and sync-error plot stay aligned. The format page adds a practical note: large recordings stream best from local disk, so download the file before opening it rather than viewing it over a slow link.

Because a recording is one file per episode, sharing an episode with a colleague is a file transfer, not a pipeline change. The site does not state file sizes, so plan storage from the Spec Pilot batch rather than from a published figure.

Yes, visually. The plots/sync_error entity shows the measured residual per frame against the 2.0 ms ceiling, and the sync-error guide says the Rerun recording plots sync error on its own timeline next to the camera and hand entities. The site states that every active stream is hardware-triggered onto one PTP clock domain and that the cross-stream ceiling is 2.0 ms.

If you would rather recompute than look, the sync-error guide says both the HDF5 and the Rerun exports carry the per-stream nanosecond timestamps, so you can recompute the residual yourself rather than trust a published number. The guide on measuring sync error shows how the residual is calculated.

The two checks are complementary: the recording is the quick look, and the timestamps are the arithmetic. For a first review of a new batch, most of the value is in watching the sync-error plot while the hands move.

They are five exports of the same QA-passed episodes, so the choice is about what you want to do with the data, not about data availability. Rerun is for looking; the other four are for loading. HDF5 is the most complete and is what Firsthand’s own QA tooling reads. RLDS suits TensorFlow-based pipelines, LeRobot suits imitation-learning pipelines and is the smallest on disk, and WebDataset suits streaming training beyond local disk.

They also treat depth and pose differently. In HDF5, RLDS and WebDataset, depth and hand joints are arrays a loader reads. In LeRobot, depth is not embedded in the mp4 and hand and body pose are flattened into observation.state. In Rerun, the pretraining-formats guide puts it as depth and hands surfaced visually on the shared timeline, for inspection rather than loading into a training loop.

You can order the .rrd alone or alongside any of the others. The site states each as a separate export of the same captured and annotated episodes, so name the combination in the spec. The fourth table below sets the five side by side.

Privacy is handled before any format is produced. Every episode is linked to a signed contributor release under a documented consent chain, with consent artefact IDs listed per episode in the batch data card. Faces and screens in frame are detected and irreversibly blurred at ingest, with a reviewer confirming the result before anything ships, and clips that cannot be de-identified are discarded rather than delivered.

Because QA and anonymization happen once, against the canonical episode, every export inherits them, including the .rrd. The head-camera footage you scrub through in the viewer is the same anonymized footage as in every other format.

The site does not state that consent records are embedded inside the .rrd file. They travel in the batch data card and the reject log that ship with the batch, so keep those alongside the recordings when you share an episode internally.

Paid, consented contributors perform the real task against your spec while wearing the capture rig. The instrumented tier is a head-mounted stereo rig: first-person RGB, a metric depth stream recovered from the calibrated stereo rig, and 6-DoF IMU data at 200 Hz from head- and wrist-mounted sensors. The rig is calibrated before the session and again after it. We do not capture robot or teleoperation data, and we do not capture at a customer’s own facility.

QA happens once, against the canonical episode, before any format-specific export is produced, so the recordings inherit the same calibrated, synced, QA-passed data as every other format. An episode is accepted only if RMS reprojection stays under 0.28 px and the drift is within tolerance. Otherwise it goes to the reject log, which ships with every batch. Only footage that clears the signed accept criteria is billed as a validated hour, and rejected items are recaptured at our cost.

You receive the dataset under a perpetual, irrevocable, worldwide commercial licence that is non-exclusive by default, with the buyer owning the resulting dataset and exclusivity available. The same terms apply whichever format the episodes are exported into.

The delivery is the .rrd recordings, one per episode, and the reject log for the batch, with a data card recording provenance, jurisdictions and the consent and anonymization records behind it. Where you also name a training format, the same episodes arrive in it.

Where coverage for the requested tasks and regions is already deep, the median time from signed spec to first validated batch is a median of 48 hours. Where it is thin, turnaround runs three to four weeks while recruiting expands in parallel across more regions. Choosing Rerun adds no separate stage, because the recordings are built from episodes that already passed QA.

Write down the tasks, environments, objects and camera tier your model needs. Then name the entities you want to see in the viewer: the head camera, hands, point cloud and sync-error plot are the ones the format page lists, and anything beyond those, such as wrist views or a particular label, is something to ask for explicitly. State the Rerun version you run, and say whether the .rrd is the only delivery or sits alongside a training format.

Most orders begin with a Spec Pilot: a small, fast first batch that proves the spec and the accept criteria hold up on real footage before you scale. On that pilot, open an episode in the viewer and check three things: the entities you depend on are present, the sync-error plot stays inside the 2.0 ms ceiling, and the hand skeleton tracks the footage.

Anything not itemised on the format pages, including file sizes and the contents of the point cloud, is agreed in the spec before volume is committed. The fifth table below shows what each spec element decides and where it shows up.