Egocentric Unboxing and Unpacking Video Data

Unpacking is a sequence of small physical decisions. A person finds the seam or the flap, cuts or tears it open, lifts the contents out, moves aside whatever was protecting them and sets the items down where they belong. The task catalog describes that work in plain terms: cut open cartons, load the stocking cart and fill shelves; open returned parcels, inspect and scan items; peel open packs and arrange instruments; open a carton and stage stock.

A first-person camera records that sequence from the viewpoint a robot policy would act from. The operator controls the view by leaning in, tilting a carton toward the head and looking inside before reaching, so the video shows not only what was done but what could be seen when it was done. Hand pose and contact events record the handling, and action segments mark where opening ends, where removal begins and where setting down starts.

It is worth being exact about what this data is. It records a person unpacking: the views, the handling and the order of steps. It is not a library of packaging types or products. The site does not state item identity, SKU, package type, deformable-material labels or task-completion labels as standard deliverables, and this page does not promise them. A buyer who needs any of them defines them in the written spec before capture begins.

Packing and kitting is the reverse direction. That page covers packing cartons, bagging and wrapping items, sealing and labeling, and building kits from parts. Unpacking starts with the closed container and ends with the items out and set down, so the first thing a policy has to learn is how to get into the container, which packing data never shows.

Pick and place is a primitive: reach, grasp, transport and place. It appears inside unpacking, because lifting an item out and setting it down are pick and place, but unpacking adds the opening, the removal of inserts and the handling of the container itself. Object rearrangement is a task family about many objects moved until a scene matches a goal arrangement, with a written start state and goal arrangement. Unpacking has a different start state, a closed container, and its goal is items out and ready for the next step rather than a target layout.

Logistics and fulfillment data covers the whole flow from receiving and putaway through picking, packing, sortation and loading, and the warehouse and retail robotics pages cover those sectors. Unpacking appears in all of them, in receiving, returns and restocking, but this page cuts across sectors and organizes the spec, the examples and the labels around the opening and removal loop. The table near the end sets the neighbouring pages side by side.

The task catalog already lists unpacking work across several environments. In retail it includes unpacking and stocking shelves with a box cutter and stocking cart (cut open cartons, load the stocking cart and fill shelves in the aisle), shelf restocking and facing from cartons, backroom organization that breaks down cartons, and a retail floor set row, unbox and stage stock: open a carton and stage stock.

Warehouse rows cover returns processing, where returned parcels are opened and items are inspected, scanned and routed, and receiving inbound pallets with a manual pallet jack. Other rows open a pack or a product: produce crates, product demos, sterile and surgical instrument packs, cans and jars, incoming mail, and packages and bags at a mock security checkpoint. The first table below lists every row.

The site does not list a row dedicated to discarding or recycling packaging after unpacking. The closest stated wording is breaking down cartons, in backroom organization. A buyer who needs packaging disposal as its own step names it in the spec, and the tables on this page show only what the catalog states.

The catalog places these tasks in a grocery and supermarket, the back of house and front of house of a retail store, a retail floor set with shelving, checkout, rail and a pack bench, a warehouse and distribution center, an office, a wet lab and clinical bench, a veterinary clinic, a working kitchen set and a mock security checkpoint. Each task row carries a status that says how soon capture can begin.

The site defines those statuses plainly. Ready to deploy means contributors and site access are set up and capture starts once a spec is signed. Active collection means recording is running now, so a spec can join an existing capture stream. Available on request means the work is scoped and recruited to spec when a buyer asks for it. In the current catalog, unpack and stock shelves with a box cutter, shelf restocking and facing, backroom organization, returns processing and receiving inbound pallets are marked Ready to deploy. The remaining unpacking rows are Available on request, and none is currently marked Active collection.

The site does not say that any of this capture takes place inside a particular customer facility, and this page does not claim it. The catalog names environment types, and where a collection happens is agreed in the spec.

The written spec is the contract, so it is worth being specific about the parts of unpacking the site leaves to the buyer. Name the containers in scope, such as cartons, parcels, bags, sterile packs, crates or cans, and the items inside them. Name the tools the operator uses; the catalog mentions a box cutter, a stocking cart, a manual pallet jack and a can opener, and a spec can name others.

Name the start state and the end state. A closed carton on a bench and an opened parcel with items scanned and routed are different goals, and the accept criteria should say which one counts as a finished episode. Name what happens to the packaging, whether it is left in place, broken down or set aside, because the site does not state a standard for that.

Finally, name the capture mode: a head-mounted view of real work on site, or a staged set with handheld grippers for demonstrations that must map onto a robot end effector.

The catalog names two capture modes. In the first, a head-mounted phone records the first-person view of real work on site, and a multiview rig adds synchronized third-person angles when the spec asks for them. In the second, a staged working set is recorded by a multiview rig while the operator performs the task with handheld grippers, so demonstrations map onto a robot end effector.

A head-mounted camera keeps the container, the hands and the station in frame from the viewpoint a manipulation policy will see from, with a wrist-mounted camera available alongside it for close-range detail when that matters more than the full-arm view. Depth, 3D hand pose, contact events and action segments are aligned to video on one hardware clock, synced under 2 ms. Distance-invalid pixels beyond the depth sensor reliable range are flagged rather than interpolated.

The site does not state a camera count, resolution, frame rate or lighting setup specific to unpacking, so those are agreed in the written spec rather than assumed. That includes how the inside of a deep carton is lit, which matters for any task where the contents are only visible from above.

Each episode follows the same episode schema used across Firsthand custom collections: ten streams synced under 2 ms, with head and wrist RGB, depth, 3D hand pose with per-joint visibility, contact and grasp events, and frame-accurate action segments. Warehouse collections also state instance segmentation alongside action segments and grasp and contact events; the site does not state it specifically for unpacking tasks.

For unpacking, hand pose tracks the operator through the cut, tear, pull, lift and set-down motions, contact events mark when fingers make and break contact with the container or an item, and action segments mark the start and end of each step, such as opening, removing and setting down. Because the labels share one clock with the video and depth, a step boundary lines up with the exact frame where it happens.

Object labels use the terms your spec sets: the skill spec decides which object classes are in scope before capture, so labels follow the vocabulary your training stack expects rather than a generic taxonomy.

Item identity, SKU, package type, deformable-material labels for cardboard, film or fabric, packaging-state labels such as sealed, cut or open, and task-completion labels are not stated anywhere on the site as standard labels. Some catalog task names mention checking contents against a manifest or scanning items, but those describe what the operator does, not a label delivered with the episode.

A per-episode success or failure label is also not stated. If your policy needs any of these, they are defined in the written spec as annotation depth, and the quote covers them before capture begins. The page does not promise them by default.

Every episode links to a signed contributor release through a documented consent chain, and the consent record is part of the delivery. Contributors are the people who perform the unpacking tasks, so their release covers the footage captured of their hands and workspace.

Faces, plates and screens are detected and irreversibly blurred at ingest, and a reviewer confirms the result before anything ships. The anonymization record is delivered with the data, so a buyer can see what was blurred and when. That matters at a pack bench or receiving dock, where coworkers, scanner screens and monitors can enter the frame.

Packaging labels, shipping addresses and barcodes are not listed as blur categories on the site, so this page does not claim they are blurred. Unpacking puts printed labels in close view by nature. If label text or addresses in frame matter for your use, raise it in the spec so the handling is agreed before capture rather than assumed afterwards.

Delivery formats are RLDS, LeRobot and WebDataset, plus HDF5, zarr and Rerun .rrd. The first three are the training-loader formats; HDF5 and zarr carry arrays for custom pipelines, and Rerun .rrd gives a multi-stream synchronized playback file for review, which suits visually checking an unpacking episode before writing a loader.

Converter source is readable, so you can retarget the schema to your own loader, and the episode schema page lists the fields. Unpacking episodes use the same schema as every other Firsthand collection, so a loader that already reads another Firsthand dataset reads an unpacking dataset without changes.

You describe the containers, items, stations, tools and end states your policy needs. We turn that into a written spec covering tasks, environments, capture tier, annotation depth and accept criteria, then quote reach, timeline and price against that document before any capture begins.

The collection runs through the same seven-stage end-to-end custom collection pipeline as every Firsthand program: contributor recruiting, capture, per-batch QA, anonymization and licensed delivery. Quality is measured against signed accept criteria, billing is per validated hour, a reject log ships with every batch, and rejected items are recaptured at our cost.

Where coverage is already deep, the first validated batch lands in a median of 48 hours from signed spec. Thin environments typically take three to four weeks, because vetting operators there is the bottleneck, and we quote that timeline up front. The catalog status on each task row tells you which case you are in.

On-demand top-ups run against the same spec and accept criteria after each training run, so new unpacking data stays consistent with what you already hold. A Spec Pilot, a small and fast first batch, can prove the spec and accept criteria before you scale volume.

Every custom collection ships under a perpetual, buyer-owned commercial license, and exclusivity is available so the data is never resold. The license is documented alongside the consent chain, so a policy trained on the data can go into a product.