Stereo Grocery Checkout and Bagging Data

It is real-world, first-person video of a person doing checkout work, recorded with two calibrated cameras so that depth can be recovered, together with 3D hand pose and the other streams the site already lists. The catalog describes the work in its own words: scan items, take payment, bag groceries by weight and hand them to the customer, and, on a retail floor set, bag groceries by weight and fragility or scan items and bag by type. The buyer sets the task and the accept criteria; Firsthand handles contributors, capture, QA, consent and delivery.

The data is human capture by paid, consented contributors, not robot or teleoperation data. This page is about grocery checkout robot training data in the sense of what a policy can learn from watching a skilled person scan and bag, and it makes no claim that any robot has been trained on it or that any robot scans or bags. People reach this topic through several phrasings, such as stereo bagging video, an egocentric supermarket cashier dataset or custom retail data collection at scale. They describe one request from different angles.

Two rows are the core of this page. In the Grocery & Supermarket environment of the Retail & Consumer Goods sector the catalog lists "Checkout and bagging", rated Easy, with the description "Scan items, take payment, bag groceries by weight and hand them to the customer." On the Retail Floor Set (shelving, checkout, rail, pack bench) it lists "Bag groceries", rated Medium, and "Scan + bag at checkout", rated Easy. Cart collection, also Easy, sits in the grocery environment and is context only.

The grocery environment examples read "Supermarket, convenience store, wholesale club, produce market, pharmacy retail floor", and the floor set examples read "Shelving, a checkout, a garment rail and a pack bench". All of these rows have the status Available on request. That means the work is scoped and recruited to spec when a buyer asks for it, so this page claims no completed collection, no delivered hours and no delivered stereo footage for any of them. The first table below reproduces the rows exactly, with three nearby checkout rows from other environments listed as context.

Not stated. The catalog lists a head-mounted phone or a multiview rig for the two grocery rows, and a multiview rig with handheld grippers for the two floor set rows. None of them lists a stereo pair. Stereo applies here only through the site general stereo offer: the site describes a stereo head rig and a hard-environments stack, and a stereo pair for a checkout order is a choice made in the skill spec and confirmed on a Spec Pilot.

This is why the page is written as a stereo and container page for these rows rather than as a description of existing stereo checkout footage. If you need stereo, say so in the brief. If you are happy with the catalog setup, the egocentric grocery bagging page describes that non-stereo route for the checkout and bagging rows.

This page is stereo capture of grocery checkout scanning and bagging. The egocentric grocery bagging and checkout page covers six checkout and bagging rows from several environments as first-person video, with stereo not its subject. The stereo retail shelf restocking page is stereo, but it covers nine restocking and facing rows on the other side of the transaction, getting product onto the shelf, and it does not cover scanning or bagging.

The egocentric shopping cart and basket retrieval page covers gathering carts and baskets and returning them to the bay or entrance, a store-associate job away from the counter, and it is not stereo. Two other pages a reader might expect, a dedicated egocentric checkout and cashier page and an egocentric retail checkout page, do not exist on the site, so they are not compared. The second table below puts the existing pages side by side with their status as the site lists it.

The catalog words the rows as short ordered jobs. Checkout and bagging is scan, then take payment, then bag by weight, then hand over to the customer. Bag groceries names one decision rule for the placement, weight and fragility. Scan + bag at checkout is scan, then bag by type. In each case the order of steps and the rule that decides where an item goes are the learnable content.

What a spec can do is cut each job into steps the buyer names and give each a start and an end. Action segments carry start_ns, end_ns, verb and noun, so one transaction becomes a sequence of labeled spans instead of one long clip, and contact events mark when a hand meets an item, the bag or the counter. The site names no scan count, no bag count and no basket size for any row, and whether one episode is a single item, a full bag or a whole basket is Not stated, so a spec that needs it sets it.

The site describes two stereo configurations. The stereo head rig, dataset 06, is a dual fisheye pair at 1080p per eye on a 62 mm baseline, with per-unit calibration, a 9-axis IMU and hand-tracking output, shipped as MCAP. The hard-environments stack is a 4K/60 head camera with a 120° lens, two 1080p/60 wrist cameras, depth at 848×480 and 30 fps and a 200 Hz 6-DoF IMU. Which of these a checkout order uses is Not stated and is agreed in the skill spec.

Whichever rig is chosen, calibration is measured every session and shipped with the episode, with a 0.28 px RMS reprojection gate. Hand pose is 21 joints per hand with a per-joint visibility flag, and the language data is one task instruction per episode. The site lists two catalog capture modes. A head-mounted phone records the first-person view of real work, with a multiview rig adding synchronized third-person angles when the spec asks for them. On a staged set, a multiview rig records the operator working with handheld grippers, so demonstrations map onto a robot end effector. The two grocery rows list the first mode and the two floor set rows list the second, so which applies to a batch is settled in the spec. Whether capture runs in a live store, on a staged set or elsewhere is Not stated, and no store is named.

Hand occlusion of the item is the first case. During a scan the fingers close around the item and hide it, and a bag held open hides what has already been placed. The site states that the hands occlude the work and that fingers disappear behind objects. The per-joint visibility flag on the 21-joint hand pose marks joints that are not visible instead of guessing them, contact events mark when a hand meets an item or the bag, and a spec can ask for wrist or multiview angles so a grip the head camera loses is still recorded.

Reflective packaging is the second. The site states that depth degrades on reflective surfaces, which per-frame confidence maps handle by masking low-confidence regions. How depth behaves on thin, translucent or crinkled bag film or on a specific kind of packaging is Not stated, so a Spec Pilot checks it on real footage. Deformable bags are the third: the site states that masks for deformable objects are shipped per frame rather than propagated, with instance masks stored at 848×480, and what counts as the object, the item, the bag or the load, is a spec decision.

Fast hand-offs are the fourth. All streams share one hardware clock with a 2.0 ms ceiling, and action segments and contact events mark the moment of the hand-over, but the site states no hand-off speed for any row. A crowded counter is the fifth. The site states that an undocumented bystander is a reject criterion, and says nothing about counter crowding or queue length as such. Store lighting is the last: the site states no standard lighting values for these rows, so conditions are fixed in the skill spec for the place in question. Dropped items and mis-placed objects are captured and action-segmented alongside clean takes on the pick-and-place page, and a spec can ask for the same here.

Every batch is validated against signed accept criteria before it ships, with a reject log. The site states that an episode is rejected when the manipulated object is out of frame for more than 15 percent of it, when more than 35 percent of depth pixels in the manipulation region are invalid, when a sync breach runs beyond the ceiling for more than three consecutive frames, when an undocumented bystander appears, or when the episode does not match the named skill spec. Episodes that fail are rejected and logged per batch, not delivered with a caveat.

Billing is per validated hour, and rejected items are recaptured at our cost. The site does not state an accuracy figure for step labels, and it lists no label for item identity, bag fill, fragile items or scanner reads. What counts as a pass for a step boundary is set in the accept criteria.

Every episode links to a signed contributor release through a documented consent chain, and the consent record is part of the delivery. Faces, plates and screens are detected and irreversibly blurred at ingest, and a reviewer confirms the result before anything ships, with the anonymization record delivered alongside. Footage that cannot be de-identified is discarded, and an undocumented bystander is a stated reject criterion. How customers and staff at a counter are separately handled is Not stated beyond that, so raise it in the spec if it matters.

One catalog row names a payment step. This page describes no payment-card handling and states no field for payment data: payment steps are handled per the site privacy pages only, which are the privacy page, the anonymization page and the DPA. The buyer owns the dataset under a perpetual, irrevocable, non-exclusive-by-default licence, with exclusivity available, and the executed agreement controls.

Firsthand offers MCAP delivery. MCAP is an open-source container format with its own official site at mcap.dev, a specification at mcap.dev/spec and a repository at github.com/foxglove/mcap; read those for what the container is and how it stores records, timestamps and attachments. This page describes the container only from those sources and does not state anything more about it.

The streams the site already states for a stereo capture are the stereo pair, depth with confidence maps, 21-joint hand pose, IMU, contact events, action segments, per-frame instance masks and calibration, with one task instruction per episode. Topic names, message schemas, encoding and rates are Not stated on the site and are agreed in the skill spec. Only dataset 06 is stated as shipped as MCAP; for any custom order, MCAP can be the only delivery or sit beside RLDS, LeRobot, WebDataset, HDF5 or zarr. The third table below maps each stream to what the site states.

Store formats, self-checkout or staffed lanes, lane layout, scanner, scale and terminal models, bag types and sizes, item counts per transaction, basket sizes, task times, throughput, error rates, force or tactile data, item identity, bag-fill or fragile-item labels and any safety or compliance certification are not stated on the site as standard fields. No store, retailer, client or brand is named for any of these rows.

If your policy needs any of these, they are written into the skill spec as annotation depth and covered in the quote before capture begins.

The site off-the-shelf catalog lists about 324,800 hours across 18 datasets, including about 64,800 hours of stereo video in 9 datasets. None of its dataset descriptions names checkout, scanning or bagging, so no off-the-shelf set stands in for these rows. One stereo set is published as a format and rig example, and it is not coverage of any checkout row.

Stereo checkout collection to a spec is therefore custom collection: custom collection at scale through a network of thousands of capture providers, for a consistent, recurring data flow to labs. Where coverage is deep, the median time from signed spec to first validated batch is a median of 48 hours; where it is thin, turnaround runs three to four weeks. All of these rows are Available on request, so the scoped-to-spec timeline applies rather than the ready-now case, and the quote states it up front.

Describe the row you want, the steps you expect to see and where each starts and ends. Say whether one episode is a single item, a full bag or a basket, what counts as the object, what counts as a reject, and whether you want a stereo pair, wrist views or a multiview rig. Then name the streams you want in the file, the topic naming and message encoding your reader expects, whether calibration ships inside the file or beside it, and whether MCAP is the only delivery.

Most programs begin with a Spec Pilot, a small first batch that proves the spec and accept criteria on real footage before you scale. The last table below shows what each spec element decides.