Egocentric Shopping Cart and Basket Retrieval Data

Egocentric cart and basket retrieval data is first-person video of a store associate collecting shopping carts and hand baskets and bringing them back, recorded from their own point of view with synchronized sensor streams alongside. The camera sits on the head, close to what a robot’s onboard camera would see, so the frames show what the hands are doing with the cart handle, the basket and the cart bay. The same idea sits behind any cart retrieval robot training data or shopping basket handling video dataset request, applied here as a custom collection scoped to your tasks.

The data is human capture: a paid, consented contributor wears the rig and performs a task from a written skill spec. It is not robot or teleoperation data. It is also not fixed-camera store footage, because the viewpoint is the person doing the work.

The task catalog lists two rows about carts and baskets directly, in the Retail & Consumer Goods sector. "Cart collection" sits in the Grocery & Supermarket environment, which the catalog illustrates with supermarkets, convenience stores, wholesale clubs, produce markets and pharmacy retail floors. It is described as collecting carts from the parking lot and store entrance and pushing them back to the cart bay. It is rated Easy and Available on request.

"Cart & Basket Retrieval" sits in the Retail Store -- Back of House environment, illustrated with the stockroom, receiving dock and backroom of supermarkets, apparel and general retail. It is described as gathering carts and baskets from the floor and car park and returning them to the entrance. It is rated Easy and Ready to deploy.

Both rows use one capture mode, a head-mounted phone with a multiview rig. The site states no cart types, cart or basket counts, store formats, store chains, distances travelled, cycle times or regions.

Two rows mention a stocking cart but are about filling shelves. "Shelf restocking" in Grocery & Supermarket is described as opening cartons, facing products forward and filling gaps from a stocking cart. "Unpack and stock shelves using a box cutter and stocking cart" in Retail Store -- Back of House is described as cutting open cartons, loading the stocking cart and filling shelves in the aisle. The cart is a tool in those rows, and the task is stocking, so this page does not count them as cart retrieval.

Checkout, bagging, produce handling and price tagging are separate rows with their own pages or environments. The grocery bagging page covers scanning and bagging, and the grocery put-away page covers putting groceries away. They are linked below.

No. No catalog row names nesting or stacking carts into a train, pushing long lines of carts, wheel or brake inspection, cart repair or cleaning, sanitizing handles, helping a customer with a cart or loading a customer’s car. The coverage table below sets each of these against what the catalog actually lists.

If a program needs a task the catalog does not name, it is written into the skill spec, and whether it can be captured is confirmed when the spec is quoted rather than assumed.

Each attempt is delivered as one episode. On this site the standard episode carries action segments, 3D hand pose for both hands with a per-joint visibility flag, contact events that mark when a hand meets an item, an objects list, one task instruction, and per-frame instance masks stored at the depth resolution of 848×480. The capture setup table below lists the stated streams and sync figures.

For cart work, an object in the list might be a cart, a basket or the cart bay, as written in the skill spec. Grip force is not a stated stream.

Action segments divide an episode into start- and end-timestamped spans, each carrying start_ns, end_ns, verb and noun. For cart work, a segment might be one basket picked up, one cart gripped or one cart pushed to the bay, as written in the skill spec. The site lists no label for the outcome of a step, such as whether a cart reached the bay or the entrance was left clear.

Validation runs against signed accept criteria written into the spec before capture, and a reject log ships with each batch. A human reviewer checks every episode. Only footage that clears the criteria is billed as a validated hour.

Delivery is in the format your stack already reads: RLDS, LeRobot, WebDataset, HDF5, zarr or Rerun .rrd, or raw streams with metadata, with readable converter source alongside. The license is perpetual, irrevocable and worldwide, buyer-owned and non-exclusive by default, with exclusivity available as a separate term. The executed agreement controls.

Retrieval looks simple in the catalog: gather, push, return. The difficulty sits in the handling. A cart is a large wheeled object that is moved by a handle, and the hands often grip it while the body is walking. A hand basket is carried, set down and picked up in different poses. The handle or rim can hide the fingers, which is why hand pose carries a per-joint visibility flag, why contact events mark when the hand meets the item, and why masks are recorded per frame. The challenge table below maps each difficulty to the stream or label this site states.

The wearer also travels between the floor, the car park and the entrance. The site lists an IMU and body pose as streams but states nothing about surfaces, slopes, weather or lighting, so the table says so.

The catalog describes the head-mounted phone as recording the first-person view of real work. The site also states that Firsthand does not capture at a customer’s own facility. Which store layout a program uses is agreed in the skill spec.

Every contributor signs a written release, and each delivered episode links to it through consent artefact IDs. At ingest, automated detection finds faces, plates and screens and blurs them irreversibly, and speech is scrubbed of names, addresses and card numbers. A human reviewer confirms that no un-consented person is identifiable in any frame, and the anonymization record ships with the data. That matters for store work, where shoppers and parked cars can appear near the entrance.

Parking-lot safety, safety certifications and protective equipment are not stated for these environments, and this page makes no such claim.

From signed spec to first validated batch, the published timeline is a median of 48 hours where coverage is deep and three to four weeks where it is thin and contributors need to be recruited first. The catalog marks "Cart & Basket Retrieval" Ready to deploy, meaning contributors and site access are already set up. "Cart collection" is Available on request, so it starts from the spec. Which range applies is confirmed when the spec is quoted.

Describe the retrieval task and its order, the carts or baskets involved, where they are gathered and returned, what counts as a finished step, and the failure or variation cases your pipeline is missing. If you need something beyond the two rows, such as nesting carts or cart repair, say so up front. That becomes a written skill spec, quoted for reach, timeline and price before any capture begins. Our skill spec guide walks through how to write one.

Most programs begin with a Spec Pilot, a small, fast first batch that proves the spec and the accept criteria on real footage before volume scales. Collections are billed per validated hour.