MCAP vs HDF5 for Robotics Training Data

Neither format is better in the abstract, because they were built for different jobs. MCAP, per mcap.dev, is an open source container file format for multimodal log data that stores multiple channels of timestamped, pre-serialized messages. HDF5, per the HDF Group, is a file format, a data model and the software to work with them, where a file holds groups and datasets. A lab choosing a delivery container for stereo egocentric data is really choosing a reader: message-oriented tooling built around channels and log time, or array-oriented tooling built around groups, datasets and slices.

This page is a decision guide, not a benchmark. The MCAP and HDF Group documentation read for it state no read speeds, write speeds, compression ratios or file sizes for either format, and the Firsthand site states none for its deliveries, so none appear here. Where a detail is missing the page says Not stated. Firsthand is not affiliated with Foxglove, the MCAP maintainers or The HDF Group.

Firsthand offers MCAP delivery, and it offers HDF5 and zarr delivery, so the choice is yours to make from your own stack rather than something dictated by data availability. The sections below set out what each format is, how it holds time-stamped multi-stream data, what tooling the official documentation lists, and when each one tends to fit.

The mcap.dev home page describes MCAP as an open source container file format for multimodal log data, supporting multiple channels of timestamped pre-serialized data and ideal for pub/sub or robotics applications. It is a Foxglove project, and the foxglove/mcap README calls it a modular, performant, serialization-agnostic container file format under the MIT License. Serialization-agnostic means the container records and replays binary messages in formats such as Protobuf, DDS (CDR), ROS and JSON. The container does not decide what a message looks like.

The home page lists the design choices that matter to a data buyer. The design is row-oriented and append-only, to minimize disk I/O and reduce the risk of data corruption during unclean shutdowns. Schemas are stored alongside the data, so files stay readable as a codebase evolves. An optional index allows efficient seeking, even over a low-bandwidth connection, and chunk-based compression with LZ4 or Zstandard is optional. The page also says MCAP is the default log format in ROS 2 and is used in production by companies from autonomous vehicles to drones.

The specification adds the file identity. The registered media type is application/vnd.mcap and the extension is .mcap. A file begins and ends with magic bytes, with a Header record first and a Footer record last, and an optional summary section supports fast lookup.

The HDF5 introduction describes HDF5 as a file format for storing HDF5 data, a data model for logically organizing and accessing that data from an application, and the libraries, language interfaces and tools for working with the format. An HDF5 file can be thought of as a container that holds a variety of heterogeneous data objects, and the two primary objects in the data model are groups and datasets. The HDF Group describes itself as a non-profit that develops and maintains the HDF5 technology suite, and says people have relied on its open-source HDF software for over 25 years.

Groups organize objects much like directories do, with a root group and path names such as /foo/zoo. A dataset holds the raw data values together with metadata: a datatype, a dataspace describing the layout and rank, and properties. A dataspace can have fixed or unlimited dimensions, and it can also select a portion of a dataset for I/O. The documentation says the default storage layout is contiguous and can be changed to chunked, or chunked and compressed, for better performance. Attributes are small named values attached to objects, typically user metadata. They do not support partial I/O and cannot be compressed or extended.

The software is written in C with optional wrappers for C++, FORTRAN and Java, and the introduction uses the Python h5py interface in its examples. It lists command-line tools including h5dump and the HDFView browser. A licence for HDF5 was not on the pages read, so it is Not stated here.

MCAP is built around the timestamped message. Per the specification, a Channel record defines an encoded stream of messages on a topic, a Schema record defines a schema, and a Message record is one timestamped message on a channel with a log_time and a publish_time. Timestamps are uint64 nanoseconds since a user-understood epoch, such as the Unix epoch or robot boot time. Because every stream is a channel and every sample is a message, several streams share one log-time axis in one file by construction.

HDF5 has no built-in notion of a message or a topic. Streams are datasets in groups, and time is whatever the writer stores. The introduction shows a dataset with attributes named Time and Pressure, and a dataspace gives the shape. A multi-stream recording in HDF5 is therefore a layout convention: which groups, which dataset per stream, and where each stream keeps its timestamps. That convention is the schema, and whoever writes the file has to document it.

This is the structural difference that matters for a delivery decision. In MCAP the container defines channels, schemas and per-message times, and the encoding of each message is left to you. In HDF5 the container defines groups, datasets, datatypes and attributes, and the meaning of the layout is left to you. Neither is inherently better at holding a stereo pair. They place the burden of convention in different places.

For MCAP, mcap.dev lists native reader and writer libraries in C++, Go, Python, Rust, Swift and TypeScript, and the repository README adds a CLI to inspect, merge and split files, installable with brew install mcap. Writers can configure optional chunking, indexing, CRC checksums and compression to trade off for their application. MCAP is also the default log format in ROS 2, which matters if your robotics stack already records logs that way.

For HDF5, the introduction describes a library written in C with wrappers for C++, FORTRAN and Java, plus high-level APIs for images, tables, dimension scales and packet tables, and examples using the Python h5py interface. The command-line tools include h5dump, which displays the contents of a file, and HDFView, a browser for HDF files. The programming model is open, access, close, with a file opened before any dataset in it.

Neither source counts third-party tools, reports adoption numbers or compares itself with the other, so this page does not rank the two ecosystems. The practical check is simple: confirm that your reader language and your viewer are on the list for the format you choose.

The site states a set of streams for Firsthand capture. The stereo head rig, off-the-shelf dataset 06, is dual fisheye at 1080p per eye with a 62 mm baseline, per-unit calibration, a 9-axis IMU and hand-tracking output, and it is stated as shipped as MCAP. Hand pose is 21 joints per hand with a per-joint visibility flag. Stereo collection lists stereo RGB, depth, IMU and audio, with calibration measured every session and shipped with the episode. Cross-stream sync is hardware-triggered onto one clock, with a median of 1.12 ms and a 2.0 ms ceiling, and drift is written to metadata.

The site’s HDF5 and zarr export is stated against the canonical episode schema: one group per episode with streams/, annotations/ and calibration/ groups. Streams include head_rgb, wrist_l/r, depth_mm, depth_conf and three IMU streams, and annotations include hand_joints. The site does not state that the stereo pair of dataset 06 appears in that HDF5 layout, and it does not state which streams are channels in an MCAP file or how they are named. Topic names, message encodings and rates are agreed in the skill spec and confirmed on a Spec Pilot. The second table above sets this out stream by stream.

Per the site’s format page, an HDF5 delivery is one .h5 or .zarr group per episode, chunked time-major in 30-frame chunks with blosc-zstd level 5, with int64 nanosecond timestamps per stream on a PTP domain. The groups mirror the episode schema: streams/, annotations/ and calibration/, and the calibration group carries both the opening and closing solves for drift checks. The site calls HDF5 the format Firsthand’s own QA tooling reads, so it is the most complete export, suited to random access into long episodes and to verifying sync yourself.

The format page also gives read guidance. Read whole 30-frame chunks rather than single frames to avoid decompression overhead, and treat zarr as the better choice for object storage and HDF5 as the better choice for a single local file. File and shard sizes are set per program in the skill spec, so no sizes appear here. The 30-frame, blosc-zstd choice is Firsthand’s own layout. The HDF Group documentation says only that a dataset layout can be chunked, or chunked and compressed.

Pick MCAP when your tooling already reads it or reads ROS 2 logs, since mcap.dev says it is the default log format in ROS 2. Pick it when you want several streams on one log-time axis with schemas inside the file, when you replay or inspect data with the mcap CLI, or when you want an append-only design meant to limit corruption after an unclean shutdown. It also fits when the consumer is an existing pipeline that processes messages in time order rather than slicing arrays.

Pick it with eyes open. MCAP is a container of messages, not an episode schema. The site does not state how episode boundaries, action segments or contact events appear in an MCAP file, so if you need them inside the file, say so in the spec. If your training loop expects RLDS, LeRobot or WebDataset, MCAP is not a training format on its own, so name the training format alongside it.

Pick HDF5 when your code is array-first: a data loader that wants a window of frames or a hand-joint array by path. The HDF Group documentation describes dataspaces that select a portion of a dataset and layouts that can be chunked. The Firsthand format page states HDF5 as the choice for random access into long episodes and for verifying sync yourself, and as the format its QA tooling reads. Pick it when you want annotations and calibration in the same file as the streams, in named groups, and when a single local file suits your workflow, with zarr as the object-storage twin.

Pick it with eyes open. The layout is a convention. HDF5 does not define a topic or a message, so a reader needs the episode schema to know what streams/head_rgb means. Per the documentation, attributes are typically small and cannot be compressed or extended, so a large block of metadata belongs in a dataset rather than an attribute.

No. The site states MCAP as one packaging option among several, and says a custom order is packaged in the format you name, such as MP4 with JSON or CSV, MCAP, LeRobot or a custom layout. The same captured and annotated episodes are stated to ship as RLDS, LeRobot, WebDataset, HDF5 and zarr, with Rerun .rrd for inspection. For MCAP, the site does not state that the file is exported from the same canonical episodes, so confirm that in the spec.

One option is to specify both: a log-oriented container for replay and inspection, and an array-oriented or training-oriented format for your loaders, with the stream list for each written down. Rerun .rrd is for inspection rather than training. The RLDS vs LeRobot comparison covers the training-format side, including which of the two embeds depth.

The first failure is assuming the format fixes the schema. MCAP leaves message encoding to you and HDF5 leaves the group and dataset layout to you, so write down topic names and encodings, or group and dataset paths, before the first batch. The second is an unindexed MCAP file. The specification says messages outside chunks cannot be indexed and may not be found by readers using the index, so ask for chunked and indexed files if you need lookup by time or topic. The site does not state whether its MCAP files use chunking, compression, CRC checksums or a summary section, so ask.

The third is reading single frames from a chunked HDF5 file instead of whole chunks, which the site’s format page warns causes decompression overhead. The fourth is assuming a stream is present. Confirm the stream list on a Spec Pilot, and remember the site states that the LeRobot layout does not embed depth. The fifth is trusting sync without checking how it maps. The site states a 2.0 ms ceiling on one hardware-triggered clock, but how that appears in MCAP message times is not stated, so verify it on a sample. The sixth is treating a rig example clip as the delivered file.

Off-the-shelf coverage is what is already in stock. The stereo category lists four stereo datasets, and only dataset 06 is stated as MCAP. Custom collection covers the rest: custom collection at scale through a network of thousands of capture providers, for a consistent, recurring data flow to labs. Where coverage is deep, the median time from signed spec to first validated batch is 48 hours, and thin coverage runs slower. Choosing MCAP or HDF5 adds no separate stage that the site states.

Tasks are grounded in the live catalog and the stereo collection pages, which cover crews, heavy machinery, surface finishing, layout marking and taping, floor mopping and surface quality inspection. The compare template cannot embed clips. The only stereo clip on the site is on the stereo sample page and is a rig example: head-mounted stereo with hands in frame. It is not an MCAP file, not an HDF5 file and not a named task. Request a sample to see the delivered files.

Name the streams you want in the file and, for MCAP, the topic names and message encoding your reader expects, or for HDF5, the group and dataset paths. State whether calibration ships inside the file or beside it, and which timestamp convention you need. For MCAP, say whether you want chunking, an index, CRC checksums and compression. For HDF5, say whether you want HDF5 or zarr and whether the default 30-frame layout suits your loader.

Say how episodes map to files, whether action segments and contact events must be inside the container, and whether a training format such as RLDS, LeRobot or WebDataset should ship beside it. Then confirm all of it on a Spec Pilot, a small first batch paid per validated hour, with a reject log that ships with every batch. The data is real human egocentric capture, never synthetic, collected by consented contributors and anonymized at ingest, and the licence is perpetual, irrevocable and buyer-owned, non-exclusive by default.

The sources read give no speeds, compression ratios or file sizes for either format, so this page makes no performance claim. The HDF5 licence was not on the pages read. The HDF5 documentation read was the introduction page, so deeper features are not covered here. For Firsthand, the site does not state MCAP topic names, message schemas, encodings or rates, which streams are channels in an MCAP file, whether an MCAP file is exported from the same canonical episodes as the other formats, or whether the stereo pair of dataset 06 appears in the HDF5 layout. Each of those is agreed in the skill spec and confirmed on a Spec Pilot.