Egocentric Box and Parcel Handling Video Data

Egocentric box and parcel handling data is real-world, first-person video of a person moving boxes and parcels through a working day: stacking them, building and wrapping pallets, carrying them to a dock or a door, taping cartons shut, scanning and labeling them, and sorting them to lanes or routes. It is recorded from the operator viewpoint, with depth, 3D hand pose, contact events and frame-accurate action segments aligned on one hardware clock.

A box is a simple object that still asks a lot of a policy. It has to be taken by its sides or underside, carried, set down square and often re-gripped on a different face. A first-person view keeps the load, both hands and the destination in the same frame, which is the viewpoint a robot arm with a head or wrist camera would act from.

This page treats box and parcel handling as a task family in its own right. Packing and kitting is about filling a container, unboxing is about opening one, and picking is about single items. Here the container is the object being handled: it is stacked, moved, sealed, scanned and sorted.

The task catalog already lists box and parcel work in several sectors. In warehouse and distribution settings it includes palletizing boxes in a stable pallet pattern, depalletizing to a shelf, moving pallets and boxes to the dock with a hand truck, staging pallets and parcels and loading them into a trailer in sequence, scanning parcels and sorting them to lanes, and applying labels to outbound boxes before sorting them by carrier or route.

In manufacturing it includes packing finished goods into cartons at the end of the line, taping cartons shut, building a pallet in layers and stretch-wrapping the load. In last-mile delivery and courier settings it includes scanning and collecting parcels at pickup, loading a vehicle, sorting parcels in a van, carrying them to a door and handing them to the recipient, and sorting letters and parcels by route at a sorting bench. Tote work is listed as well, from nesting and stacking empty totes to inducting totes onto a conveyor, and a retail back room row moves cartons from the backroom to the floor.

A custom spec is scoped from that same vocabulary. You describe the boxes, parcels, stations and sequence your policy needs, and the written spec covers them, with the catalog tasks as a starting point rather than a fixed menu. The first two tables below list every row with its description, environment and status as the catalog states them.

Pallet work is the clearest box handling task in the catalog. Palletize boxes is described as stacking boxes in a stable pallet pattern, and depalletize to shelf as removing from a pallet and shelving. Both sit in a warehouse zone with a conveyor, racking, pallets and a pack bench, are recorded with a multiview rig and handheld grippers, and are Available on request.

In manufacturing, three Ready to deploy rows cover the end of the production line: assembly line packaging, which packs finished goods into cartons and seals and stacks them; boxing and palletizing finished goods, which builds the pallet in layers and wraps it for shipping; and a boxing and palletizing row that uses tape guns and stretch wrap. In the warehouse, loading outbound freight moves pallets and boxes to the dock with a hand truck and stacks and stretch-wraps the load, while loading outbound trucks stages pallets and parcels at the dock and loads them into the trailer in sequence.

The warehouse pages also name pallet stacking and shrink-wrap, carton break-down and a two-person long-item carry as skills. Those are named skills on the warehouse pages rather than rows in the task catalog, so a spec that needs them states them explicitly.

Parcel work in the catalog centers on the scan, the label and the lane. Scan and sort parcels is described as scanning parcels and sorting them to lanes. Sort and label outbound boxes prints labels, applies them to outbound boxes and sorts the boxes by carrier or route, using a label printer and box cutter. Labeling applies product and barcode labels to units or cartons by hand or with a label gun, and weigh and seal weighs, verifies and seals a package.

Last-mile and mail rows add the hand-over. Courier pickup and drop-off scans and collects parcels, loads the vehicle and hands them over. Parcel delivery sorts parcels in the van, carries them to the door, scans them and hands them to the recipient. Postal sorting and mail handling sort letters and parcels by route. All of these are Available on request except the warehouse label-and-sort row and the manufacturing labeling row, which are Ready to deploy.

Barcode scanning and label application are covered in more depth on their own page, linked below. Here the scan is one step in moving a parcel from one place to the next.

The catalog places these rows in a warehouse and distribution center, in a warehouse zone with a conveyor, racking, pallets and a pack bench, in a plastic and light-assembly manufacturer, in last-mile delivery and courier work, in the back of house of a retail store and in an office and professional services setting for mail handling. Each task row carries a status that says how soon capture can begin.

The site defines those statuses plainly. Ready to deploy means contributors and site access are set up and capture starts once a spec is signed. Active collection means recording is running now, so a spec can join an existing capture stream. Available on request means the work is scoped and recruited to spec when a buyer asks for it. In the current catalog the warehouse and distribution center rows, the manufacturing rows and the retail back-of-house row are Ready to deploy, while the warehouse zone rows, the last-mile rows and mail handling are Available on request.

The site does not state which specific sites a box and parcel collection runs at, so the environment, access and schedule are agreed in the written spec.

The catalog names two capture modes. In the first, a head-mounted phone records the first-person view of real work on site, and a multiview rig adds synchronized third-person angles when the spec asks for them. In the second, a staged working set is recorded by a multiview rig while the operator performs the task with handheld grippers, so demonstrations map onto a robot end effector.

A head-mounted camera keeps the lift, carry and place motion in frame from the viewpoint a manipulation policy will see from, with a wrist-mounted camera available alongside it for close-range detail. Depth, 3D hand pose, contact events and action segments are aligned to video on one hardware clock, synced under 2 ms. Distance-invalid pixels beyond the depth sensor reliable range are flagged rather than interpolated.

The site does not state a camera count, resolution or frame rate specific to box handling, and it does not state a load height or a pallet size, so those are agreed in the written spec rather than assumed.

Each episode follows the same episode schema used across Firsthand custom collections: ten streams synced under 2 ms, with head and wrist RGB, depth, 3D hand pose with per-joint visibility, contact and grasp events, and frame-accurate action segments. Warehouse collections also state instance segmentation alongside action segments and grasp and contact events.

For box and parcel handling, hand pose tracks the operator through the grip, lift, carry and place, contact events mark when the fingers make and break contact with a box, parcel or tote, and action segments mark the start and end of each step in the sequence. Because the labels share one clock with the video and depth, a step boundary lines up with the exact frame where it happens.

Object labels use the terms your spec sets. The skill spec decides which object classes are in scope before capture, so labels follow the vocabulary your training stack expects rather than a generic taxonomy.

Box or parcel identity, box weight, size or dimensions, parcel counts or throughput, carrier or shipping-label data, address text, barcode or SKU values, force or tactile data, a stack-stability or task-completion label and any ergonomic or safety certification are not stated anywhere on the site as standard fields. Some catalog task names mention scanners, labels or weighing, but those describe what the operator does, not a field delivered with the episode.

If your policy needs any of these, they are defined in the written spec as annotation depth, and the quote covers them before capture begins. The page does not promise them by default, and it makes no warehouse safety or ergonomics claim.

Delivery formats are RLDS, LeRobot and WebDataset, plus HDF5, zarr and Rerun .rrd. The first three are the training-loader formats; HDF5 and zarr carry arrays for custom pipelines, and Rerun .rrd gives a multi-stream synchronized playback file for review. The format table below sets each against what its official documentation states.

Converter source is readable, so you can retarget the schema to your own loader, and the episode schema page lists the fields. Box and parcel episodes use the same schema as every other Firsthand collection, so a loader that already reads another Firsthand dataset reads this one without changes.

Every batch is validated against signed accept criteria before it ships, with a reject log, and delivery is rolling, per validated batch.

Every episode links to a signed contributor release through a documented consent chain, and the consent record is part of the delivery. Contributors are the people who perform the box and parcel tasks, so their release covers the footage captured of their hands and workspace.

Faces, plates and screens are detected and irreversibly blurred at ingest, and a reviewer confirms the result before anything ships. The anonymization record is delivered with the data. A dock, a van or a doorstep can put other people in frame, which is why that pass matters here.

Parcel and shipping labels, address text and barcodes are not listed as blur categories on the site, so this page does not claim they are blurred. If text on a parcel matters for your use, raise it in the spec so the handling is agreed before capture.

Where coverage is already deep, the first validated batch lands in a median of 48 hours from signed spec. Thin environments typically take three to four weeks, because vetting operators there is the bottleneck, and we quote that timeline up front. The catalog status on each task row tells you which case you are in.

On-demand top-ups run against the same spec and accept criteria after each training run, so new box and parcel data stays consistent with what you already hold. The site does not state a batch size or a later-batch turnaround, so those are quoted against your spec.

You describe the boxes, parcels, stations and sequence your policy needs. We turn that into a written spec covering tasks, environments, capture tier, annotation depth and accept criteria, then quote reach, timeline and price against that document before any capture begins.

A Spec Pilot, a small and fast first batch, can prove the spec and the accept criteria before you scale volume. The collection runs through the same seven-stage end-to-end custom collection pipeline as every Firsthand program: contributor recruiting, capture, per-batch QA, anonymization and licensed delivery. Billing is per validated hour, a reject log ships with every batch, and rejected items are recaptured at our cost.

Every custom collection ships under a perpetual, buyer-owned commercial license, and exclusivity is available so the data is never resold. The license is documented alongside the consent chain, so a policy trained on the data can go into a product.