Egocentric Truck Loading and Unloading Video Data

Egocentric truck loading and unloading data is real-world, first-person video of a person moving goods between a dock and a vehicle. In the site task catalog that means staging pallets and parcels at the dock and loading them into the trailer in sequence, and unloading inbound goods from the truck, checking quantities against the paperwork and moving them to storage. It is recorded from the operator viewpoint, with depth, 3D hand pose, contact events and frame-accurate action segments aligned on one hardware clock.

The boundary between dock and vehicle is where a lot of manipulation happens. A load has to be carried to an edge, placed into a bounded space and, on the way in, taken out of one and moved on. A first-person view keeps the load, both hands and the destination in the same frame, which is the viewpoint a robot arm with a head or wrist camera would act from.

The box and parcel page covers the handled object, the pallet page covers pallets and the package delivery page covers the last mile. Here the focus is the loading and unloading sequence.

The task catalog lists three rows that name the dock and the truck or trailer directly, and a fourth that unloads inbound pallets without naming a vehicle. All four sit in the Warehouse and Distribution Center environment, are rated Medium and are Ready to deploy. Loading Outbound Trucks stages pallets and parcels at the dock and loads them into the trailer in sequence. Receiving and Unloading unloads inbound goods from the truck, checks quantities against the paperwork and moves them to storage.

Two more rows sit beside them. One moves pallets and boxes to the dock with a hand truck and stacks and stretch-wraps the load, which is the step before a load goes onto a vehicle. The other breaks down inbound pallets with a pallet jack, checks the contents against the manifest and shelves them, which is the step after goods come off one. The first table below lists each row with its description, environment, skill rating and status exactly as the catalog states them.

A custom spec is scoped from that same vocabulary. You describe the docks, loads and sequence your policy needs, and the written spec covers them, with the catalog tasks as a starting point rather than a fixed menu. Nearby rows that load a vehicle or mention a dock without naming a truck or trailer, such as the courier rows, are kept out of the first table and listed in the second, with the reason for each.

The four truck, dock and pallet rows are listed under the Warehouse and Distribution Center environment, which the catalog describes with examples of an e-commerce fulfillment hub, a distribution center, a cold storage facility and a freight warehouse. A retail back of house environment also lists a receiving dock in its examples, but its task rows are about stocking, so they are not counted here.

The site defines the statuses plainly. Ready to deploy means contributors and site access are set up and capture starts once a spec is signed. Active collection means recording is running now, so a spec can join an existing capture stream. Available on request means the work is scoped and recruited to spec when a buyer asks for it. All four truck, dock and pallet rows are Ready to deploy.

The catalog describes this as a working environment captured on site, not a staged set. The site does not state which specific sites a collection runs at, and it does not state capture at a buyer own dock or fleet, so the environment, access and schedule are agreed in the written spec.

Every truck, dock and pallet row names the same capture mode: a head-mounted phone records the first-person view of real work on site, and a multiview rig adds synchronized third-person angles when the spec asks for them. The site does not list a handheld-gripper variant for these rows.

A head-mounted camera keeps the carry, place and unload motion in frame from the viewpoint a manipulation policy will see from, with a wrist-mounted camera available alongside it for close-range detail. Depth, 3D hand pose, contact events and action segments are aligned to video on one hardware clock, synced under 2 ms. Distance-invalid pixels beyond the depth sensor reliable range are flagged rather than interpolated.

The site does not state a camera count, resolution or frame rate specific to loading, and it does not state a dock height or trailer dimension, so those are agreed in the written spec rather than assumed.

Each episode follows the same episode schema used across Firsthand custom collections: ten streams synced under 2 ms, with head and wrist RGB, depth, 3D hand pose with per-joint visibility, contact and grasp events, and frame-accurate action segments. Warehouse collections also state instance segmentation alongside action segments and grasp and contact events.

For loading and unloading, hand pose tracks the operator through the grip, carry and place, contact events mark when the fingers make and break contact with a pallet, parcel or carton, and action segments mark the start and end of each step in the sequence. Because the labels share one clock with the video and depth, a step boundary lines up with the exact frame where it happens.

Object labels use the terms your spec sets, decided before capture.

A loading task is a sequence of manipulations that ends at a bounded space, and a policy has to learn both the individual moves and the order. First-person data gives the hand and object detail for each move, and the action segments give the order. That combination is what the site describes for every task family it collects.

The view also matters. A robot with a head camera and a wrist camera acts from a viewpoint close to the one a head-mounted capture records, which is why the site records both a head and a wrist stream. A humanoid or a mobile manipulator working a dock has the same need to see the load, its hands and the destination together.

The site states no robot results for truck loading and makes no claim that any model was trained on this data. This page describes what is collected and how it is delivered, not what a policy trained on it will do.

Delivery formats are RLDS, LeRobot and WebDataset, plus HDF5, zarr and Rerun .rrd. The first three are the training-loader formats; HDF5 and zarr carry arrays for custom pipelines, and Rerun .rrd gives a multi-stream synchronized playback file for review. The format table below sets each against what its official documentation states.

Converter source is readable, so you can retarget the schema to your own loader, and the episode schema page lists the fields. Loading episodes use the same schema as every other Firsthand collection, so a loader that already reads another Firsthand dataset reads this one without changes.

Every batch is validated against signed accept criteria before it ships, with a reject log, and delivery is rolling, per validated batch.

Every episode links to a signed contributor release through a documented consent chain, and the consent record is part of the delivery. Contributors are the people who perform the loading and unloading tasks, so their release covers the footage captured of their hands and workspace.

Faces, plates and screens are detected and irreversibly blurred at ingest, and a reviewer confirms the result before anything ships. The anonymization record is delivered with the data.

Paperwork text, manifest text and labels on parcels are not listed as blur categories on the site, so this page does not claim they are blurred. If text on a document or parcel matters for your use, raise it in the spec so the handling is agreed before capture.

Where coverage is already deep, the first validated batch lands in a median of 48 hours from signed spec. Thin environments typically take three to four weeks, because vetting operators there is the bottleneck, and we quote that timeline up front. The catalog status on each task row tells you which case you are in.

On-demand top-ups run against the same spec and accept criteria after each training run, so new loading and unloading data stays consistent with what you already hold. The site does not state a batch size or a later-batch turnaround, so those are quoted against your spec.

You describe the docks, loads and sequence your policy needs. We turn that into a written spec covering tasks, environments, capture tier, annotation depth and accept criteria, then quote reach, timeline and price against that document before any capture begins.

A Spec Pilot, a small and fast first batch, can prove the spec and the accept criteria before you scale volume. The collection runs through the same seven-stage end-to-end custom collection pipeline as every Firsthand program: contributor recruiting, capture, per-batch QA, anonymization and licensed delivery. Billing is per validated hour, a reject log ships with every batch, and rejected items are recaptured at our cost.