MCAP vs LeRobot Format for Robot Training Data

They answer different questions, so the choice is about where the data goes next. MCAP, per mcap.dev, is an open source container file format for multimodal log data, with multiple channels of timestamped pre-serialized messages. LeRobotDataset, per the Hugging Face docs, is a standardized format for robot learning data that pairs tabular signals with multi-camera video and metadata. If you are logging and replaying sensor streams, MCAP is the natural container. If you are training in the LeRobot library, LeRobotDataset is the layout it reads.

This page is a decision guide for labs choosing how stereo egocentric training data is delivered. It is not a benchmark. The sources read for it state no read speeds, write speeds, compression ratios or file sizes for either format, and the Firsthand site states none for its deliveries, so none appear here. Where a detail is missing the page says Not stated. Firsthand is not affiliated with Foxglove, Hugging Face or the LeRobot maintainers.

Firsthand offers MCAP delivery and the LeRobot v2.1 layout. Where coverage is deep, the median to first validated batch is 48 hours, and for work the catalog does not cover yet, Firsthand runs custom collection at scale through a network of thousands of capture providers, for a consistent, recurring data flow to labs.

The mcap.dev home page describes MCAP as an open source container file format for multimodal log data, ideal for pub/sub or robotics applications. It is a Foxglove project, and the foxglove/mcap README calls it a modular, performant, serialization-agnostic container under the MIT License. Serialization-agnostic means the container does not dictate the message encoding: the home page lists Protobuf, DDS (CDR), ROS and JSON among the formats it can record and replay.

The design is row-oriented and append-only, to minimize disk I/O and reduce the risk of corruption during unclean shutdowns. Schemas are stored alongside the data. An optional index allows efficient seeking, even over a low-bandwidth connection, and chunk compression can use LZ4 or Zstandard while still supporting indexed reads. Chunking, indexing, CRC checksums and compression are optional, so a given file may or may not have them. The home page also says MCAP is the default log format in ROS 2, and the README gives the media type application/vnd.mcap.

The huggingface/lerobot README describes LeRobot as models, datasets and tools for real-world robotics in PyTorch, and says LeRobotDataset is a standardized, scalable format hosted on the Hugging Face Hub. Its structure is synchronized MP4 videos (or images) for vision and Parquet files for state and action data. The README lists tools to delete episodes, split by indices or fractions, add or remove features, and merge datasets. The repository is listed under the Apache License 2.0.

The LeRobotDataset v3.0 documentation names three pillars. Tabular data, meaning states, actions and timestamps, lives in Apache Parquet. Visual data is camera frames concatenated and encoded into MP4, sharded per camera. Metadata is JSON or Parquet describing feature names, dtypes, shapes, frame rates, normalization statistics and episode segmentation. The directory layout holds meta/info.json, meta/stats.json, meta/tasks.jsonl, meta/episodes/, data/ and videos/. The docs say image transforms are applied at training time and raw images are stored, so augmentation can change later without re-recording.

No. MCAP is a container: it holds messages on channels and says nothing about what a robot episode is. LeRobotDataset is a dataset layout: it names features like action and observation.state, ties them to episodes and tasks, and is read by a loader that returns tensors. That is why the comparison is less a rivalry than a question of which layer you need.

The practical consequence is that an MCAP file needs a mapping step before a LeRobot training run, and a LeRobotDataset has no channel model for replaying a live message stream. The site states no converter from an MCAP delivery to LeRobot, so if you need both, name both in the skill spec rather than assuming one derives from the other.

In MCAP a multi-stream recording is a set of channels on one time axis, each message carrying its timestamp, per mcap.dev. In LeRobotDataset, low-dimensional high-frequency signals such as states, actions and timestamps are stored as Parquet rows, and video frames sit in per-camera MP4 files. The loader can return temporal windows through delta_timestamps, which are seconds relative to the current frame, for example the frames 0.2 and 0.1 seconds earlier.

For Firsthand capture, the site states hardware-triggered synchronization onto one clock, with a 1.12 ms median and a 2.0 ms ceiling. In the LeRobot delivery the site states per-frame timestamps aligned to the head-camera clock. For an MCAP delivery the site does not state how that sync maps to message times, so ask for it, and verify alignment in the delivered file rather than assuming the container preserves it.

The site states the v2.1 layout: parquet plus one mp4 per camera in the Hugging Face datasets layout, loading with lerobot.common.datasets. The official docs now describe LeRobotDataset v3.0, where many episodes share each Parquet or MP4 file and episode boundaries come from metadata. They say v3.0 is included in lerobot 0.4.0 and later and ship a converter that aggregates v2.1 per-episode files into larger shards.

That gap is worth an explicit line in the skill spec. If your training stack is on v3.0, ask whether you need v3.0 or the stated v2.1 layout plus a conversion step. The site does not state a v3.0 delivery, so this page does not claim one. The version table above sets the two side by side from the sources.

The site states these streams for its stereo egocentric work: a stereo pair from dual fisheye cameras at 1080p per eye with a 62 mm baseline on dataset 06, head and wrist RGB, metric depth at 848×480 and 30 Hz, hand pose of 21 joints per hand with a per-joint visibility flag, and a 9-axis IMU on dataset 06. Calibration is measured every session and shipped with the episode. The streams table above sets each against what the site states for a LeRobot and an MCAP delivery.

For LeRobot the site is specific: features observation.images.head, observation.images.wrist_l and wrist_r, observation.state and action, with hand and body pose packed into the state vector, hands first. Depth is not embedded in the mp4, so pull it from the HDF5 export if your policy consumes it. The site does not state that the stereo pair is a LeRobot feature. For MCAP the honest gap is the file layout: the site does not state topic names, message schemas, encodings, rates or which streams are channels. Those are agreed in the skill spec and confirmed on a Spec Pilot.

Pick MCAP when you want several streams on one time axis with an optional index for seeking, when you inspect data from the MCAP CLI (installable with brew install mcap), or when your tooling already reads ROS 2 logs, since mcap.dev calls MCAP the default log format in ROS 2. Native libraries exist in C++, Go, Python, Rust, Swift and TypeScript, and the repository is MIT licensed. MCAP is a container of messages, not an episode schema, so if you need episode boundaries or action segments inside the file, say so in the spec.

Pick LeRobot when your policies train with the LeRobot library (pip install lerobot, with extras such as lerobot[dataset] and lerobot[training]) and you want LeRobotDataset returning PyTorch tensors to a DataLoader. The repository is Apache License 2.0. Video is mp4, so the site warns decoding is the bottleneck: use the dataset’s built-in video backend rather than decoding per item. Depth is not in the mp4, and datasets you record yourself need finalize() before pushing, per the docs.

No. The site states that a custom order is packaged in the format you name, such as MP4 with JSON or CSV, MCAP, LeRobot or a custom layout, and that the same captured and annotated episodes ship as RLDS, LeRobot, WebDataset, HDF5 and zarr, with Rerun .rrd for inspection. MCAP is one option among them.

The sources read document no MCAP-to-LeRobot converter, and the site states none, so treat any conversion as work to scope. The only documented LeRobot conversion is the official v2.1 to v3.0 migration. For the training-format side of the choice, the RLDS vs LeRobot comparison covers the alternatives, and the MCAP vs ROS bag and MCAP vs HDF5 pages cover the container side.

The kitchen catalog on this site lists knife work, with dice, julienne and chiffonade, noted as fine bimanual dexterity, tool grip and control. The site’s own LeRobot loading snippet is built around a kitchen knife dicing example. For that task a policy lab training with LeRobot wants the head and wrist cameras, the pose state vector and the action feature in LeRobotDataset form, with hand visibility intact for the fine grip phases.

A lab studying the same task with stereo geometry, calibration and an IMU timeline wants the stereo head rig as MCAP, where the streams stay separate channels on one clock. Those are two reads of the same skill, and the catalog rows set the actual task list. The spec states which streams matter for the skill and which format carries them.

The first failure is assuming the container fixes the schema. MCAP leaves message encoding to you, so write down topic names and encodings before the first batch. The second is a version mismatch: a loader on LeRobot v3.0 expecting file-based shards, handed a v2.1 per-episode layout. The third is losing the stream you needed: depth is not in the LeRobot mp4, and the stereo pair is not a stated LeRobot feature.

The fourth is slicing observation.state in the wrong order, since hands come before body. The fifth is trusting a conversion: the site does not state a converter in either direction, so test one on a Spec Pilot before volume. The sixth is assuming sync survived export. The site states a 2.0 ms ceiling at capture, so verify alignment in the delivered file, whichever format it uses. The seventh is an unindexed MCAP file when you need seeking by time, because indexing is optional.

Firsthand offers MCAP delivery and exports the LeRobot v2.1 layout directly. The off-the-shelf stereo head rig, dataset 06, ships as MCAP with dual fisheye cameras at 1080p per eye, a 62 mm baseline, per-unit calibration, a 9-axis IMU and hand-tracking output. Off-the-shelf coverage and custom collection work together: start from the off-the-shelf pages to see formats and streams, then specify custom collection for the skills and environments you need. Custom collection runs at scale through a network of thousands of capture providers, for a consistent, recurring data flow to labs.

The site does not state MCAP topic names, message schemas, encodings, rates or files per episode, whether its MCAP files use chunking, compression, CRC checksums or an index, or a LeRobot v3.0 delivery. Those are questions for the skill spec and a Spec Pilot. For a format example, see the stereo sample clip page, which shows rig output and is a format example, not an MCAP file or a LeRobotDataset.

Write a skill spec that names the task, the environment and the streams you need, and name the delivery format: MCAP, LeRobot, another format the site lists, or several. If your loader is on a specific LeRobot version, say so. Confirm the layout on a Spec Pilot, then pay per validated hour. Where coverage is deep, the median to first validated batch is 48 hours.

For work that needs scale, custom collection runs at scale through a network of thousands of capture providers, for a consistent, recurring data flow to labs. Use the Request a custom dataset link below, or read the stereo MCAP hub page and the LeRobot format page first for the delivery detail.