Egocentric Stereo Data in MCAP Format

It is first-person video from two calibrated cameras, with hand pose and sensor streams, written to .mcap files. MCAP is the container and stereo is the capture. They are separate choices, so this page treats the format and the rig separately.

The site states MCAP delivery in two places. The off-the-shelf stereo head rig, dataset 06, ships as MCAP, and a custom order is packaged in the format you name, such as MP4 with JSON or CSV, MCAP, LeRobot or a custom layout. The site does not state topic names, message schemas, encodings, rates or files per episode. Those are agreed in the skill spec.

Firsthand is not affiliated with Foxglove or the MCAP maintainers. The facts about MCAP below come from mcap.dev and the foxglove/mcap repository, linked in the sources.

MCAP is an open source container file format for multimodal log data, supporting multiple channels of timestamped pre-serialized data. It is serialization-agnostic, so messages can be Protobuf, DDS (CDR), ROS, JSON and more. Its design is row-oriented and append-only, it stores schemas alongside the data, and it offers an optional index plus optional LZ4 or Zstandard chunk compression. mcap.dev also describes it as the default log format in ROS 2.

Reader and writer libraries exist for C++, Go, Python, Rust, Swift and TypeScript, and the mcap CLI can inspect, merge and split files. The repository is MIT-licensed; that is the MCAP project’s licence, not the licence of a Firsthand delivery.

A file begins and ends with magic bytes. Between them sit a Header record, a data section and a Footer, with an optional summary section. The data section holds Schema, Channel and Message records, plus Chunk, Message Index, Attachment and Metadata records. A Message record carries a channel ID, a log time, a publish time and the data, and timestamps are uint64 nanoseconds since a user-understood epoch.

Attachment records hold auxiliary artifacts, and the specification names calibration data as an example. The registered media type is application/vnd.mcap with the .mcap extension. Whether a Firsthand file uses chunking, compression, CRC checksums or a summary section is not stated on the site; ask for it in the spec.

For dataset 06 the site states a calibrated stereo head rig with hand-tracking output and 9-axis IMU, shipped as MCAP. Site-wide, Firsthand hand pose is 21 joints per hand with a per-joint visibility flag. Stereo collection lists stereo RGB, depth, IMU and audio, with calibration measured every session and shipped with the episode.

What is not stated is which of those streams are channels in an MCAP file, how they are named or at what rate. Depth, wrist cameras, action segments and contact events are all in that category. If a stream matters to you, name it in the skill spec and confirm it on a Spec Pilot batch rather than assuming it. The first two tables below set out what is stated and what is not.

The site states RLDS, LeRobot, WebDataset, HDF5/zarr and Rerun .rrd as exports of the same captured and annotated episodes. MCAP is stated as a packaging option for custom orders and for dataset 06, but the site does not say that an MCAP file is exported from the same canonical episodes, so confirm that in the spec.

The practical difference is the reader. MCAP files are read with the MCAP libraries and CLI, while RLDS, LeRobot and WebDataset are read by their own loaders. If your training loop expects one of those, name it in the spec alongside MCAP. The third table below sets them side by side.

Off-the-shelf covers what is already in stock: the stereo category lists four stereo datasets, and only dataset 06 is stated as MCAP. Custom collection covers the rest: custom collection at scale through a network of thousands of capture providers, for a consistent, recurring data flow to labs.

Where coverage for the requested tasks and regions is already deep, the median time from signed spec to first validated batch is a median of 48 hours. Where it is thin, turnaround runs three to four weeks. Choosing MCAP adds no separate stage that the site states.

Write down the tasks, environments, objects and camera tier your model needs. Then name the streams you want in the file, the topic naming and message encoding your reader expects, whether calibration ships inside the file or beside it, and whether MCAP is the only delivery or sits beside a training format.

Most orders begin with a Spec Pilot, a small first batch that proves the spec and the accept criteria on real footage before you scale. Paid, consented contributors perform the real task, faces and screens are blurred at ingest, and rejected episodes go to a reject log that ships with every batch. The fourth table below shows what each spec element decides.