Stereo Heavy Machinery Operation Data

It is first-person video of a person operating machine controls, recorded from their own point of view with two calibrated cameras and with hand pose and sensor streams alongside. The task is the one catalog row the site lists: operating the levers and control panels of presses, cranes or machinery from the operator station. Stereo describes the capture, and MCAP describes the container. They are separate choices.

The site does not state that stereo is already running on this row. The catalog lists a head-mounted phone or multiview rig as the capture setup, and a stereo pair is a capture choice agreed in the skill spec and confirmed on a Spec Pilot. Firsthand is not affiliated with Foxglove or the MCAP maintainers.

The egocentric heavy machinery operation data page covers the task: the catalog row, the neighbouring rows that are not counted, the labels and the privacy steps. This page does not repeat that. It covers the two choices layered on top of the same single row: a calibrated stereo pair for depth, and MCAP as the delivery container.

Both pages describe the same row and make the same limited claims. Neither names a machine brand, a model, a facility or a region, and neither states an operator licence, a safety record or an autonomy result.

One row counts: Operating Heavy Machinery Controls. It sits in the Heavy Industry & Fabrication environment, is rated hard, lists the capture setup Phone (head-mounted) / multiview rig, and is marked available on request rather than ready to deploy. The first table below reproduces it as the catalog states it.

The description names presses, cranes and machinery in one row. The site does not list a separate row for each, so a spec that needs only one of them is settled in a Spec Pilot rather than assumed.

The hard-environments page states that stereo keeps working outdoors where active depth sensors struggle, and that many programs capture both and sync them on one clock. Whether an operator station is indoors or outdoors is not something the catalog row states, so that is agreed in the spec.

Stereo depth is recovered from the calibrated pair rather than from a standalone depth sensor, and depth is delivered with confidence maps that mask low-confidence regions. That matters at a control station, where a lever, a handle or a glossy panel can give uncertain depth. The site gives no depth accuracy figure for this row.

The site describes two stereo setups and does not say which one a machinery order uses. The hard-environments stack is a global-shutter 4K/60 head camera with a 120° lens, two 1080p/60 wrist cameras, a depth sensor at 848×480 and 30 fps, and 200 Hz 6-DoF IMUs. The off-the-shelf stereo head rig, dataset 06, is dual fisheye cameras at 1080p per eye, a 62 mm baseline, per-unit calibration, a 9-axis IMU and hand-tracking output.

Which rig, resolution and streams a machinery order uses is Not stated on the site. The second table below sets what is stated beside what is not. Intrinsics and extrinsics are re-solved at the start and end of every session, and streams are hardware-triggered onto one clock with a 2.0 ms cross-stream ceiling.

MCAP is an open source container file format for multimodal log data, supporting multiple channels of timestamped pre-serialized data. It is serialization-agnostic, so messages can be Protobuf, DDS (CDR), ROS, JSON and more. Its design is row-oriented and append-only, it stores schemas alongside the data, and it offers an optional index plus optional LZ4 or Zstandard chunk compression.

Reader and writer libraries exist for C++, Go, Python, Rust, Swift and TypeScript, and the mcap CLI can inspect, merge and split files. The foxglove/mcap repository is MIT-licensed; that is the MCAP project’s licence, not the licence of a Firsthand delivery. The MCAP hub page and the glossary entry cover the format in full.

Firsthand offers MCAP delivery, and dataset 06 is stated as shipping as MCAP. For machinery work the site states hand pose, a stereo-derived depth stream, IMU, contact events and action segments elsewhere on the site. What it does not state is which of those are channels in an MCAP file, how they are named, what encoding they use or at what rate.

No topic name, message schema or rate is stated on this page, and none should be assumed. Name the streams that matter in the skill spec and confirm them on a Spec Pilot batch. The third table below sets each stream beside what the site states about it.

Hands spend much of a controls task behind or over the controls, and fingers disappear behind objects in manipulation. The visibility flag on each joint is the answer the site gives: an occluded finger is flagged, not guessed. A spec can also ask for wrist views so a grip that the head camera loses is still recorded.

Calibration is the second risk, because stereo depth is only as good as the rectification behind it. The site states calibration at the start and end of every session under a 0.28 px RMS gate. If a person who is not a contributor appears in frame, faces and screens are blurred at ingest, a reviewer confirms the result, and footage that cannot be de-identified is discarded.

The site does not describe the station itself: whether it is a live machine, a trainer or a mock-up, how many controls it has, or how a control is labeled. It lists no control position, machine state, load or lever-travel label and no contact force reading. Rejected episodes go to a reject log that ships with every batch.

Off-the-shelf covers what is already in stock: the stereo category lists the stereo datasets and hand-tracking video, and only dataset 06 is stated as MCAP. No off-the-shelf dataset is stated to cover heavy machinery controls. Custom collection covers the rest: custom collection at scale through a network of thousands of capture providers, for a consistent, recurring data flow to labs.

Where coverage is deep, the median time from signed spec to first validated batch is a median of 48 hours; where it is thin, turnaround runs three to four weeks. The site does not say which applies to heavy industry, so ask when you request the quote. Collections are billed per validated hour, and capture at your own facility is listed as not offered.

Describe the control task, the kind of machine, the setting and the variation your pipeline is missing. Then name the streams you want in the file, the topic naming and message encoding your reader expects, whether calibration ships inside the file or beside it, and whether MCAP is the only delivery or sits beside RLDS, LeRobot or WebDataset.

Most programs begin with a Spec Pilot, a small first batch that proves the spec and accept criteria on real footage before you scale. The fourth table below shows what each spec element decides.