Stereo Warehouse Order Picking Data

It is first-person video of a person picking customer orders in a warehouse: walking the pick path with a handheld scanner, picking items to a tote and, on one row, packing and taping the order. It is recorded from their own point of view with two calibrated cameras, with hand pose and sensor streams alongside. The tasks are the rows the site’s catalog lists for it. Stereo describes the capture, and MCAP describes the container. They are separate choices.

The site does not state that stereo is already running on these rows. The catalog lists a head-mounted phone or a multiview rig as the capture setup, and a stereo pair is a capture choice agreed in the skill spec and confirmed on a Spec Pilot. The data is human capture by paid, consented contributors, not robot or teleoperation data. Firsthand is not affiliated with Foxglove or the MCAP maintainers.

Two, both in the Warehouse & Distribution Center environment of the Logistics & Warehousing sector. The first is "Order Picking", rated Easy: "Picking, packing, sorting, loading and inventory handling in distribution facilities." The second is named "Pick and pack customer orders using a handheld scanner and packing tape dispenser", rated Easy: "Walk the pick path with a scanner, pick items to the tote and pack and tape the order."

Both list a head-mounted phone or a multiview rig as the capture setup, and both are Ready to deploy. The site says nothing further about what that status means in volume, hours or delivered stereo footage, so this page claims none, and what applies to a given order is confirmed when you request the quote. This page names only the tools and steps that appear in the rows: picking items to orders, a handheld scanner, totes, a packing tape dispenser and packing orders. The first table below reproduces the two rows as the catalog states them.

The egocentric bin picking page treats picking items from bins into totes as egocentric capture, with its own stream picture. The stereo order packing and taping page is the stereo and MCAP treatment of four packing rows, and it shares the pick and pack row with this page, but it centers on packing, taping and labeling and does not include the "Order Picking" row. The warehouse picking dataset page describes an egocentric dataset the site lists as Shipping.

This page narrows to the two order picking rows named above and gives them their own stereo and MCAP treatment, stream map and failure modes. The comparison table below sets the pages side by side.

A pick is a reach, a grasp, a lift and a drop into a tote, with a scanner in the other hand. A stereo pair recovers depth from the calibrated frame pair rather than from a standalone depth sensor, so the shelf, the item, the tote and the hands share a frame. The site also states that depth degrades on wet and reflective surfaces and handles that with per-frame confidence maps.

The site gives no depth accuracy figure for picking, so whether a barcode, a small item or a tote edge is resolvable in depth is Not stated. Its warehouse picking dataset page states that long sight-lines push depth beyond the 4 m reliable range and that distance-invalid pixels are flagged, not interpolated; a spec should check long aisles on a Spec Pilot. Capture at your own facility is listed as not offered.

The site describes two stereo setups and does not say which one an order picking order uses. The hard-environments stack is a global-shutter 4K/60 head camera with a 120° lens, two 1080p/60 wrist cameras, a depth sensor at 848×480 and 30 fps, and 200 Hz 6-DoF IMUs. The off-the-shelf stereo head rig, dataset 06, is dual fisheye cameras at 1080p per eye, a 62 mm baseline, per-unit calibration, a 9-axis IMU and hand-tracking output.

Which rig, resolution and streams an order picking order uses is Not stated on the site, and the third table below sets what is stated beside what is not. Intrinsics and extrinsics are re-solved at the start and end of every session, and streams are hardware-triggered onto one clock with a 2.0 ms cross-stream ceiling. The catalog states that a multiview rig adds synchronized third-person angles when the spec asks for them, and wrist cameras give a closer view of the hands, which matters when one hand holds the scanner and the other takes the item.

MCAP is an open source container file format for multimodal log data, supporting multiple channels of timestamped pre-serialized data. It is serialization-agnostic, so messages can be Protobuf, DDS (CDR), ROS, JSON and more. Its design is row-oriented and append-only, it stores schemas alongside the data, and it offers an optional index plus optional LZ4 or Zstandard chunk compression.

Reader and writer libraries exist for C++, Go, Python, Rust, Swift and TypeScript, and the mcap CLI can inspect, merge and split files. The foxglove/mcap repository is MIT-licensed; that is the MCAP project’s licence, not the licence of a Firsthand delivery. The MCAP hub page and the glossary entry cover the format in full.

Firsthand offers MCAP delivery, and dataset 06 is stated as shipping as MCAP. For picking work the site states hand pose, a stereo-derived depth stream with confidence maps, IMU, contact events, action segments and per-frame instance masks elsewhere. What it does not state is which of those are channels in an MCAP file, how they are named, what encoding they use or at what rate.

No topic name, message schema or rate is stated on this page, and none should be assumed. Name the streams that matter in the skill spec and confirm them on a Spec Pilot batch. The fourth table below sets each stream beside what the site states about it.

Occlusion is the first risk. A scanner, a tote or the items around a pick can hide the fingers that hold an item. The visibility flag on each joint is the answer the site gives, contact events mark when a hand meets an item, masks are recorded per frame, and a spec can ask for wrist views so a grip the head camera loses is still recorded. The site’s warehouse picking dataset page lists mis-picks, dropped totes and jams as its failure cases.

Sequence and definition are the second. The "Order Picking" row names picking, packing, sorting, loading and inventory handling, and the site does not say which of those a given order includes. The spec decides how each step is split into action segments and whether packing and taping sit inside the episode. A spec that says only “pick the order” gives one long segment. Pick rates, SKU counts, order volumes, warehouse management system names and throughput are Not stated and are raised in the spec. Force and tactile readings are not among the listed streams of standard capture.

Calibration is the third, because stereo depth is only as good as the rectification behind it: calibration is measured at the start and end of every session under a 0.28 px RMS gate. An episode is also rejected when the manipulated object is out of frame for more than 15 percent of it, when more than 35 percent of depth pixels in the manipulation region are invalid, or when an undocumented bystander appears, and a reject log ships with every batch. The warehouse picking dataset page states that rig cabling is routed under hi-vis PPE at every site, which slightly changes wrist extrinsics, hence per-session recalibration.

The site’s warehouse picking dataset page lists multi-person scenes as the highest of any environment it describes, so other people in the aisle are a realistic case. An undocumented bystander is a stated reject criterion. How workers and other people are handled on a given site beyond that is Not stated, and this page makes no claim about it.

For every delivered episode the site states that faces, plates and screens are blurred irreversibly, and that footage that cannot be de-identified is discarded. The anonymization record ships with the data. Anything beyond that general rule is settled in the spec before capture.

The site does list an off-the-shelf warehouse picking dataset, described as egocentric, with a status of Shipping and 1,420 validated hours on its own page. That page does not state stereo capture or MCAP delivery, and this page does not claim off-the-shelf stereo coverage of order picking. The only stereo head rig the site states as shipping as MCAP is dataset 06. So stereo order picking in MCAP is custom collection: custom collection at scale through a network of thousands of capture providers, for a consistent, recurring data flow to labs. The two can sit together, with the egocentric dataset as a baseline and custom stereo capture for the variation your pipeline is missing.

Where coverage is deep, the median time from signed spec to first validated batch is a median of 48 hours; where it is thin, turnaround runs three to four weeks. The site does not say which speed applies to a stereo order picking order, so the timeline is quoted against your spec. Collections are billed per validated hour, and rejected episodes and recaptures are never billed.

Describe the tasks and where each starts and ends: the walk along the pick path, the scan, the pick to the tote, whether packing and taping are inside the episode, the start state of the tote and the shelf, what counts as a reject, and the variation your pipeline is missing. Then name the streams you want in the file, the topic naming and message encoding your reader expects, whether calibration ships inside the file or beside it, and whether MCAP is the only delivery or sits beside RLDS, LeRobot or WebDataset.

Most programs begin with a Spec Pilot, a small first batch that proves the spec and accept criteria on real footage before you scale. The fifth table below shows what each spec element decides.