Stereo Retail Shelf Restocking Data

It is first-person video of a person restocking shelves, recorded from their own point of view with two calibrated cameras and with hand pose and sensor streams alongside. The tasks are the rows the site’s catalog lists across its retail environments. Stereo describes the capture, and MCAP describes the container. They are separate choices.

The site does not state that stereo is already running on these rows. The catalog lists a head-mounted phone or a multiview rig as the capture setup, with handheld grippers on the retail floor set rows, and a stereo pair is a capture choice agreed in the skill spec and confirmed on a Spec Pilot. The data is human capture by paid, consented contributors, not robot or teleoperation data. Firsthand is not affiliated with Foxglove or the MCAP maintainers.

Nine, across four environments under Retail & Consumer Goods. In Grocery & Supermarket, whose listed examples include a supermarket, a convenience store, a wholesale club, a produce market and a pharmacy retail floor, "Shelf restocking" is described as opening cartons, facing products forward and filling gaps on shelves from a stocking cart, and "Produce handling" as unpacking produce crates, trimming and culling items, stacking displays and misting or rotating stock. Both are rated Easy and both are listed as Available on request.

In Retail Store -- Back of House, "Shelf Restocking & Facing" is described as refilling shelves from cartons and pulling products to the front edge with labels facing out, "Unpack and stock shelves using a box cutter and stocking cart" as cutting open cartons, loading the stocking cart and filling shelves in the aisle, and "Retail Stocking Back of House" as moving cartons from the backroom to the floor and stocking shelves and displays. All three are rated Easy and are listed as Ready to deploy. In Retail Floor Set, "Shelf restock + face", "Unbox + stage stock" and "Restock display case" are rated Easy and listed as Available on request. In Retail Store -- Front of House, "Merchandising and facing" is rated Medium and listed as Available on request.

The site says nothing further about what each status means in volume, hours or delivered stereo footage, so this page claims none, and what applies to a given order is confirmed when you request the quote. This page names only the tools and steps that appear in the nine rows. The first table below reproduces them as the catalog states them.

The retail robotics training data page covers shelf stocking, facing, restocking and picking in retail as egocentric capture, and has no stereo-pair or MCAP treatment. The egocentric grocery put-away page is about putting groceries away at home and keeps retail shelf restocking apart as a different task. The price tagging and shelf signage page and the backroom organization page cover their own rows, not the restocking rows.

This page is the stereo and MCAP treatment of the nine restocking rows together. The off-the-shelf retail shelf stocking dataset is a separate, already-listed product; its entry states depth and 3D hand pose, and does not state a stereo pair or MCAP delivery. The comparison table below sets the pages side by side.

Restocking is about where the hands, the carton or product and the shelf edge sit relative to one another: how far a pack is pulled to the front edge, where a gap is filled, how a carton is cut open. A stereo pair recovers depth from the calibrated frame pair rather than from a standalone depth sensor, so the carton, the product, the shelf and the hands share a frame.

The site gives no depth accuracy figure and no label for a faced product, a filled gap or a full shelf, so whether those are resolvable in depth is Not stated. The site states elsewhere that wet and reflective surfaces degrade active-stereo depth. Whether retail packaging, shrink wrap or a chilled case behaves that way is Not stated, and a retail spec should expect the same care. Capture at your own facility is listed as not offered.

The site describes two stereo setups and does not say which one a retail order uses. The hard-environments stack is a global-shutter 4K/60 head camera with a 120° lens, two 1080p/60 wrist cameras, a depth sensor at 848×480 and 30 fps, and 200 Hz 6-DoF IMUs. The off-the-shelf stereo head rig, dataset 06, is dual fisheye cameras at 1080p per eye, a 62 mm baseline, per-unit calibration, a 9-axis IMU and hand-tracking output.

Which rig, resolution and streams a retail order uses is Not stated on the site, and the third table below sets what is stated beside what is not. Intrinsics and extrinsics are re-solved at the start and end of every session, and streams are hardware-triggered onto one clock with a 2.0 ms cross-stream ceiling. The catalog states that a multiview rig adds synchronized third-person angles when the spec asks for them, and wrist cameras give a closer view of the hands.

MCAP is an open source container file format for multimodal log data, supporting multiple channels of timestamped pre-serialized data. It is serialization-agnostic, so messages can be Protobuf, DDS (CDR), ROS, JSON and more. Its design is row-oriented and append-only, it stores schemas alongside the data, and it offers an optional index plus optional LZ4 or Zstandard chunk compression.

Reader and writer libraries exist for C++, Go, Python, Rust, Swift and TypeScript, and the mcap CLI can inspect, merge and split files. The foxglove/mcap repository is MIT-licensed; that is the MCAP project’s licence, not the licence of a Firsthand delivery. The MCAP hub page and the glossary entry cover the format in full.

Firsthand offers MCAP delivery, and dataset 06 is stated as shipping as MCAP. For retail work the site states hand pose, a stereo-derived depth stream with confidence maps, IMU, contact events, action segments and per-frame instance masks elsewhere. What it does not state is which of those are channels in an MCAP file, how they are named, what encoding they use or at what rate.

No topic name, message schema or rate is stated on this page, and none should be assumed. Name the streams that matter in the skill spec and confirm them on a Spec Pilot batch. The fourth table below sets each stream beside what the site states about it.

Hands are the first risk. Fingers go inside cartons and behind packs, and a carton or a stocking cart can hide the shelf being filled. The visibility flag on each joint is the answer the site gives, contact events mark when a hand meets an item, and a spec can ask for wrist views so a grip the head camera loses is still recorded.

People in frame are the second. The site’s retail shelf stocking dataset notes that every bystander is blurred at ingest and that clips where a bystander cannot be de-identified are discarded, which is why episode yield there is lower than for kitchen capture. The same entry notes that chilled-case capture is time-boxed to protect stock, so those episodes are shorter. It lists no faced, full or gap-filled label, so what counts as a finished step is written into the spec up front.

Calibration is the third, because stereo depth is only as good as the rectification behind it: calibration is measured at the start and end of every session under a 0.28 px RMS gate. An episode is also rejected when the manipulated object is out of frame for more than 15 percent of it or an undocumented bystander appears, and a reject log ships with every batch. Force and tactile readings are not among the listed streams.

For every delivered episode the site states that faces, plates and screens are blurred irreversibly, and that footage that cannot be de-identified is discarded. Who may be in frame, and how a store’s own signage, labels or staff are handled, is agreed in the spec.

The site states no planogram-compliance, throughput, accuracy, shrink or certification claim for these rows, so this page describes none.

Off-the-shelf covers what is already in stock: the retail shelf stocking egocentric dataset is listed as Shipping, with a grocery aisle, a chilled case, an apparel fixture and back-of-house staging among its environments and facing, case-cutting and shelf-loading among its skills. Only dataset 06 is stated as MCAP, and this page does not claim off-the-shelf stereo coverage of the restocking rows. Custom collection covers the rest: custom collection at scale through a network of thousands of capture providers, for a consistent, recurring data flow to labs.

Where coverage is deep, the median time from signed spec to first validated batch is a median of 48 hours; where it is thin, turnaround runs three to four weeks. The site does not say which speed applies to a stereo retail order, so ask when you request the quote. Collections are billed per validated hour.

Describe the stations and their order, the cartons, crates, stocking cart, box cutter and shelves and their start state, what counts as a finished step, and the variation your pipeline is missing. Then name the streams you want in the file, the topic naming and message encoding your reader expects, whether calibration ships inside the file or beside it, and whether MCAP is the only delivery or sits beside RLDS, LeRobot or WebDataset.

Most programs begin with a Spec Pilot, a small first batch that proves the spec and accept criteria on real footage before you scale. The fifth table below shows what each spec element decides.