Egocentric Grocery Put-Away Video Data

Egocentric grocery put-away video data is real-world, first-person video of a person putting groceries away: taking items out of bags, carrying them across a kitchen, opening a cupboard door and placing each item where it belongs. It is recorded from the operator viewpoint, with depth, 3D hand pose, contact events and frame-accurate action segments aligned on one hardware clock.

Put-away asks a lot of a manipulation policy because it is a long chain, not a single grasp. A person reaches into a bag, decides where an item goes, opens a door, makes room, places the item and closes up again. A first-person view keeps the bag, both hands and the storage space in the same frame, which is the viewpoint a robot arm with a head or wrist camera would act from.

This page treats grocery put-away as a task family in its own right. Grocery bagging covers the checkout counter, object rearrangement covers moving many objects to a goal arrangement, and unpacking covers opening a delivery. Here the task starts with items in bags and ends with items in storage.

The site states grocery put-away in one place and covers its neighbours in the task catalog. Its everyday-routines page lists Putting groceries away as a household routine, next to unloading a dishwasher, wiping counters and loading a washing machine. The task catalog has no row for it, so the routine has no catalog description, skill rating or status.

The closest catalog row is Load/unload dishwasher, in a staged Working Kitchen Set: rack dishes and unload to cupboard. Nearby rows sit in a grocery and supermarket environment, which lists checkout and bagging and shelf restocking, and in commercial kitchens, which list putting clean ware back on shelves.

A custom spec is scoped from that vocabulary. You describe the items, storage spaces, stations and sequence your policy needs, and the written spec covers them, with the catalog rows as a starting point rather than a fixed menu. The first two tables list what the site states and the nearby rows with their environment, skill rating and status.

Putting groceries away is the routine the site states most directly. It is listed as a household routine, one of four in a kitchen and household example that also covers unloading a dishwasher, wiping counters and loading a washing machine.

The site gives no item list, no number of items per episode and no storage-location list for the routine. Which groceries are used, which spaces they go into and how an episode starts and ends are agreed in the written spec rather than assumed.

Because the routine chains several steps, action segments matter. A spec can ask for the whole sequence from bag to shelf or isolate one step, and the segment boundaries mark where unbagging ends, where carrying gives way to opening a door and where placing gives way to closing it.

Load/unload dishwasher is described as racking dishes and unloading to a cupboard. It is rated Easy and its status is Available on request. It is a staged row, recorded by a multiview rig while the operator uses handheld grippers, in a Working Kitchen Set described as a stovetop, counter, sink, dishwasher, utensils and food props.

The row is about dishes, not groceries, but the motion is the one a put-away policy needs: take an item from one place, open a cupboard and place it inside. The catalog describes the gripper capture mode as one where demonstrations map onto a robot end effector, so the recorded grip is made with a tool the robot can reproduce rather than a bare hand.

The site does not state which cupboard layouts or heights are used, so those are agreed in the spec.

The site does not list a fridge, freezer or pantry task. No catalog row names a refrigerator, a freezer or a pantry shelf, and the site states no temperature data or refrigerated handling for any task. This page does not present fridge and pantry stocking as a stated catalog row.

What a buyer can do is ask for it. A spec can name the storage spaces your policy has to reach, such as a refrigerator shelf, a freezer drawer, a pantry shelf or a kitchen cabinet, and the written spec and quote cover them before any capture begins. Whether a particular space can be captured is confirmed in the spec, not assumed from this page.

The same applies to door and drawer handling. The site states hand pose and contact events for every episode, which carry the grip on a handle, but it states no door or drawer label as a standard field.

Retail shelf restocking is a different task from putting groceries away at home, and this page keeps them apart. The catalog lists Shelf restocking in a grocery and supermarket environment: open cartons, face products forward and fill gaps on shelves from a stocking cart. It is rated Easy and is Available on request.

Restocking works from a cart in an aisle toward a shelf that belongs to a store. Put-away works from a bag toward the storage spaces of a kitchen. The hands do related things, but the setting, the objects and the order of steps differ, so the site does not describe a policy trained for one as covering the other.

Retail shelf stocking has its own dataset page, linked below, and grocery checkout and bagging has its own task page.

The catalog places the nearby rows in a staged working kitchen set, a grocery and supermarket environment, commercial kitchens and a staged care room. Each task row carries a status that says how soon capture can begin.

The site defines those statuses plainly. Ready to deploy means contributors and site access are set up and capture starts once a spec is signed. Active collection means recording is running now, so a spec can join an existing capture stream. Available on request means the work is scoped and recruited to spec when a buyer asks for it. The kitchen cupboard row is Available on request, so the timeline is the scoped-to-spec case, and the grocery routine has no catalog status. The commercial kitchen rows that put clean ware back on shelves are Active collection, but they describe commercial kitchens, not home put-away.

The site names no grocery chain, retailer, appliance or brand for any row, and it does not state capture at a customer own home or facility. Environment, access and schedule are agreed in the written spec.

The catalog names two capture modes. In the first, a head-mounted phone records the first-person view of real work on site, and a multiview rig adds synchronized third-person angles when the spec asks for them. In the second, a staged working set is recorded by a multiview rig while the operator performs the task with handheld grippers, so demonstrations map onto a robot end effector.

A head-mounted camera keeps the carry, reach and place motion in frame from the viewpoint a manipulation policy will see from, with a wrist-mounted camera available alongside it for close-range detail. Depth, 3D hand pose, contact events and action segments are aligned to video on one hardware clock, synced under 2 ms. Distance-invalid pixels beyond the depth sensor reliable range are flagged rather than interpolated.

The site does not state a camera count, resolution or frame rate specific to put-away, so those are agreed in the written spec rather than assumed.

Each episode follows the same episode schema used across Firsthand custom collections: ten streams synced under 2 ms, with head and wrist RGB, depth, 3D hand pose with per-joint visibility, contact and grasp events, and frame-accurate action segments.

For put-away, hand pose tracks the operator through the grip, the carry and the placement, contact events mark when the fingers make and break contact with an item, a bag or a handle, and action segments mark the start and end of each step in the sequence. Because the labels share one clock with the video and depth, a step boundary lines up with the exact frame where it happens.

Object labels use the terms your spec sets. The skill spec decides which object classes are in scope before capture, so labels follow the vocabulary your training stack expects rather than a generic taxonomy.

Item counts per episode, object categories or SKU labels, shelf or storage-location labels, throughput, force or tactile data, temperature data, a task-completion label and any food-safety certification are not stated anywhere on the site as standard fields. A fridge, freezer or pantry task row is not stated either.

If your policy needs any of these, they are defined in the written spec as annotation depth, and the quote covers them before capture begins. The page does not promise them by default, and it makes no food-safety or certification claim.

Delivery formats are RLDS, LeRobot and WebDataset, plus HDF5, zarr and Rerun .rrd. The first three are the training-loader formats; HDF5 and zarr carry arrays for custom pipelines, and Rerun .rrd gives a multi-stream synchronized playback file for review. The format table below sets each against what its official documentation states.

Converter source is readable, so you can retarget the schema to your own loader, and the episode schema page lists the fields. Put-away episodes use the same schema as every other Firsthand collection, so a loader that already reads another Firsthand dataset reads this one without changes.

Every batch is validated against signed accept criteria before it ships, with a reject log, and delivery is rolling, per validated batch.

Every episode links to a signed contributor release through a documented consent chain, and the consent record is part of the delivery. Contributors are the people who put the groceries away, so their release covers the footage captured of their hands and workspace.

Faces, plates and screens are detected and irreversibly blurred at ingest, and a reviewer confirms the result before anything ships. The anonymization record is delivered with the data. A kitchen can put other people in frame, which is why that pass matters here.

Text on packaging and labels is not listed as a blur category on the site, so this page does not claim it is blurred. If text on a package matters for your use, raise it in the spec so the handling is agreed before capture.

Where coverage is already deep, the first validated batch lands in a median of 48 hours from signed spec. Thin environments typically take three to four weeks, because vetting operators there is the bottleneck, and we quote that timeline up front. The catalog status on each task row tells you which case you are in, and the kitchen cupboard row is currently Available on request.

On-demand top-ups run against the same spec and accept criteria after each training run, so new put-away data stays consistent with what you already hold. The site does not state a batch size or a later-batch turnaround, so those are quoted against your spec.

You describe the items, storage spaces, stations and sequence your policy needs. We turn that into a written spec covering tasks, environments, capture tier, annotation depth and accept criteria, then quote reach, timeline and price against that document before any capture begins.

A Spec Pilot, a small and fast first batch, can prove the spec and the accept criteria before you scale volume. The collection runs through the same seven-stage end-to-end custom collection pipeline as every Firsthand program: contributor recruiting, capture, per-batch QA, anonymization and licensed delivery. Billing is per validated hour, a reject log ships with every batch, and rejected items are recaptured at our cost.

Every custom collection ships under a perpetual, buyer-owned commercial license, and exclusivity is available so the data is never resold. The license is documented alongside the consent chain, so a policy trained on the data can go into a product.