Egocentric Depalletizing and Cross-Dock Relabeling Data

It is real-world, first-person video of the hand work at two points in a distribution flow, recorded with depth and 3D hand pose and the other streams the site already lists. The catalog describes the work in short phrases: remove from pallet and shelve, strip and re-label for re-route, and apply and align shipping labels. The buyer sets the task and the accept criteria; Firsthand handles contributors, capture, QA, consent and delivery.

The data is human capture by paid, consented contributors, not robot or teleoperation data. This page makes no claim that any robot has been trained on it. People reach this topic through several phrasings, such as egocentric depalletizing data, first-person cross-dock relabel video, a depalletize to shelf dataset or custom warehouse data collection. They describe one request from different angles.

Three rows in the Warehouse Zone environment name this work. Depalletize to shelf is rated Easy and reads "Remove from pallet and shelve." Cross-dock re-label is rated Easy and reads "Strip and re-label for re-route." Apply shipping labels is rated Easy and reads "Apply and align shipping labels." All three are listed with the capture setup "Multiview rig + handheld grippers" and the status Available on request.

Available on request means the work is scoped and recruited to spec when a buyer asks for it. This page therefore claims no completed collection and no delivered hours for any of the three rows. The first table below reproduces the rows exactly as the catalog lists them, with the other Warehouse Zone rows shown as nearby context.

The row is one line: remove from pallet and shelve. It has two ends, a pallet and a shelf, and a spec should say where an episode begins and ends: at the first item off the pallet, at a shelved item, or across a whole pallet. The site states no episode length for this row, no item list and no pallet or shelf dimensions, so none is described here.

The footage matters most at the moments when a hand is behind a carton face or inside a gap in the stack, and when the item is set down on the shelf. Those are the moments hand pose with visibility flags and contact events are meant to cover. The site gives no instruction here and this page offers none; it describes only what the catalog row names. The nearby Palletize boxes row, which stacks boxes in a stable pallet pattern, is the reverse direction and is a separate Medium row.

The row reads "Strip and re-label for re-route." It is the only one of the three that names a cross-dock, and the site does not say what is stripped, what the replacement label carries or how the re-route is decided. A spec should define the object, whether the carton, the label or the pair counts as the thing being handled, and how that choice shows up in contact events and instance masks.

The site states no label types, no label sizes and no reading or scanning step for this row, and it does not say whether a label is read, printed or only placed. Whether the work happens at a bench, at a stack or in the open is also not stated. These details are agreed in the skill spec, and this page describes the row only in the words the catalog uses.

The row reads "Apply and align shipping labels." Two things are named, placing the label and lining it up, and the finish of an episode is a spec decision: the moment the label touches the carton, the moment it is pressed flat, or a check that it sits in the right place. The site states no label size, no label content and no carton size for this row.

Label work is close-range and fingertip-level, so a camera that sees the hands well matters more here than elsewhere in this family. The site lists no label for label position, alignment quality or pass and fail, and it does not state a shipping label as a delivered field. This page gives no instruction on how to apply or align a label and describes only what the catalog row names.

The environment lists fourteen rows, twelve rated Easy and two rated Medium, all with the same capture setup and the status Available on request. Next to the three above sit Pack carton ("Pack a carton with void-fill and tape."), Poly-bag + tag ("Bag an item and apply a tag."), Bundle multi-item ("Kit and bundle a multi-item order."), Weigh + seal ("Weigh, verify, and seal a package."), Scan + sort parcels ("Scan parcels and sort to lanes."), Each-pick to tote ("Pick individual items to a tote.") and Load conveyor ("Load items onto a moving conveyor.").

The rest are Stack empty totes ("Nest and stack empty totes."), Tote induction ("Induct totes onto a conveyor."), Palletize boxes ("Stack boxes in a stable pallet pattern.") and Stretch-wrap pallet ("Wrap a pallet with stretch film."); the last two are the Medium rows. These are mentioned as context only. A spec that chains rows, for example a depalletize, a re-label and then a conveyor load, is scoped against the written spec, and this page does not claim any such chain has been captured.

The catalog states one mode for all three rows: a staged working set is recorded by a multiview rig while the operator performs the task with handheld grippers, so demonstrations map onto a robot end effector. The site does not name a head-mounted first-person view for these rows, so whether one is added is settled in the skill spec. The environment itself is listed as "Conveyor, racking, pallets, totes, a pack bench and scale".

Each episode follows the same episode schema as other Firsthand custom collections, with ten streams synced under a 2.0 ms ceiling, head and wrist RGB, depth, 3D hand pose with per-joint visibility, contact and grasp events and frame-accurate action segments. The site also states instance segmentation for warehouse collections. Calibration is measured every session and shipped with the episode, with a 0.28 px RMS reprojection gate. The language data is one task instruction per episode. Camera count, resolution and frame rate for the set as quoted are Not stated.

Not stated. The catalog lists no stereo pair for the three rows. Stereo applies only through the site general stereo offer, which describes a stereo head rig, dataset 06, with dual fisheye cameras at 1080p per eye on a 62 mm baseline, per-unit calibration, a 9-axis IMU and hand-tracking output shipped as MCAP, and a hard-environments stack. Whether a depalletizing or relabeling order uses either is a choice made in the skill spec and confirmed on a Spec Pilot.

If stereo is the point of the order, the stereo pages for pallets and for warehouse sorting and labeling describe stereo for different sets of rows. This page is about the catalog rows as listed and does not describe existing stereo footage of them.

The egocentric pallet building and wrapping page treats palletizing, pallet breakdown and stretch-wrapping as its main topic across several settings, and it does not name the cross-dock re-label or shipping-label rows. The egocentric box and parcel handling page covers palletizing, carrying, sealing, scanning and sorting, and Depalletize to shelf appears there as one row. The barcode scanning and labeling page treats scanning and labeling as a family across many settings, and it lists Apply shipping labels and Cross-dock re-label among many rows.

The stereo palletizing and stretch wrapping page covers two manufacturer rows for building and wrapping a pallet, the stereo warehouse receiving and truck loading page covers four Warehouse & Distribution Center rows, and the stereo warehouse sorting and labeling page covers three rows in that same environment, all Ready to deploy. This page makes the three Warehouse Zone rows the main topic. The second table below puts the pages side by side with their status as the site lists it.

Hand occlusion is the first case. A stack hides the fingers behind a carton face, and a carton held in both hands hides the label. The site states that the hands occlude the work and that fingers disappear behind objects. The per-joint visibility flag marks hidden joints instead of guessing them, contact events mark when a hand meets an item or a label, and a spec can ask for wrist or multiview angles so a grip the main camera loses is still recorded.

Reflective surfaces are the second. The site states that depth degrades on reflective surfaces, which per-frame confidence maps handle by masking low-confidence regions. How depth behaves on a glossy label, a film-covered load or a specific item is Not stated, so a Spec Pilot checks it on real footage. Deformable objects are the third: the site states that masks for deformable objects are shipped per frame rather than propagated, with instance masks stored at 848×480, and what counts as the object, the carton, its contents or the label, is a spec decision.

Crowded scenes, belt motion and lighting are the rest. Many near-identical items on a pallet or a shelf make instance separation harder, and the site states instance segmentation for warehouse collections without stating an accuracy for a given load. The environment lists a conveyor, but none of the three rows names belt work and the site states no belt speed. Streams share one hardware clock, so anything that moves stays aligned across streams. An undocumented bystander is a stated reject criterion, and no lighting values are stated for this set, so conditions are fixed in the skill spec.

Every batch is validated against signed accept criteria before it ships, with a reject log. The site states that an episode is rejected when the manipulated object is out of frame for more than 15 percent of it, when more than 35 percent of depth pixels in the manipulation region are invalid, when a sync breach runs beyond the ceiling for more than three consecutive frames, when an undocumented bystander appears, or when the episode does not match the named skill spec. Failed episodes are rejected and logged per batch, not delivered with a caveat.

Billing is per validated hour, and rejected items are recaptured at our cost. The site does not state an accuracy figure for step labels, and it lists no label for item identity, shelf position or label content. What counts as a pass for a step boundary, such as the first item off the pallet or the moment a label is placed, is set in the accept criteria.

Every episode links to a signed contributor release through a documented consent chain, and the consent record is part of the delivery. Faces, plates and screens are detected and irreversibly blurred at ingest, and a reviewer confirms the result before anything ships, with the anonymization record delivered alongside. Footage that cannot be de-identified is discarded.

The site states no rule specific to shipping labels, so blurring of label text is Not stated beyond the general rule above, and it is raised in the spec if it matters. The buyer owns the dataset under a perpetual, irrevocable, non-exclusive-by-default licence, with exclusivity available, and the executed agreement controls. The anonymization and privacy pages are linked below.

Delivery formats are RLDS, LeRobot and WebDataset, plus HDF5, zarr and Rerun .rrd. Firsthand also offers MCAP delivery. MCAP is an open-source container format with its own official site at mcap.dev, a specification at mcap.dev/spec and a repository at github.com/foxglove/mcap; this page describes the container only from those sources. Only dataset 06 is stated as shipped as MCAP, and for any custom order MCAP can be the only delivery or sit beside a training format.

Topic names, message schemas, encoding and rates are Not stated on the site and are agreed in the skill spec. Converter source is readable, and the episode schema page lists the fields, so a loader that already reads another Firsthand dataset reads this one without changes.

Carrier, marketplace, retailer, label and pallet brand names, label types and sizes, case weights, pallet and case counts, item lists, shelf layouts, throughput, task times, success and error rates, and force or tactile data are not stated on the site. None is claimed here.

If your policy needs any of the fields that can be specified, they are written into the skill spec as annotation depth and covered in the quote before capture begins.

The site off-the-shelf catalog lists about 324,800 hours across 18 datasets, including about 64,800 hours of stereo video in 9 datasets. No off-the-shelf set is described as covering depalletizing to a shelf, cross-dock re-labeling or applying shipping labels, so this is custom collection.

Custom warehouse data collection runs as custom collection at scale through a network of thousands of capture providers, for a consistent, recurring data flow to labs. Where coverage is deep, the median time from signed spec to first validated batch is a median of 48 hours; where it is thin, turnaround runs three to four weeks. All three rows are Available on request, so the scoped-to-spec timeline applies, and the quote states it up front.

Describe the rows you want and where each starts and ends. Say whether an episode is one item to a shelf, one pallet, one re-label, one shipping label, or a chain of them, which item groups are in scope, what counts as the object, what counts as a reject, and whether you want the catalog multiview rig, wrist views or a stereo pair. Then name the streams you want in the file and the delivery format.

Most programs begin with a Spec Pilot, a small first batch that proves the spec and accept criteria on real footage before you scale. The last table below shows what each spec element decides.