Egocentric Sorting Task Video Data (Recycling, Returns, Parts)

Sorting is one of the most common jobs in warehouses, back rooms, mailrooms, laundries and packing lines, and one of the hardest to show a policy well. A person picks up an item, looks at it, decides where it belongs and puts it there, then does it again. The site lists sorting tasks in many of its catalog sectors, from sorting damaged goods into markdown or disposal bins to sorting parcels to lanes.

A first-person camera records that sequence from the viewpoint a robot would act from, so the video shows not only where the item went but what could be seen when the choice was made. Hand pose and contact events record the handling, and action segments mark where each step starts and ends. A manipulation policy can learn from this evidence without any robot being involved in the capture.

It helps to be exact about what the data is. It records a person sorting: the views, the handling and the order of steps. The site does not state category labels, material-type labels, sort-correctness or accuracy labels, or object identity or pose labels, so this page does not promise them.

Object rearrangement data covers a person moving many objects from where they are to where they belong, putting items away and restoring a scene to a goal arrangement. This page treats sorting as the case in which each item is assigned to one of several destinations, such as a returns bin, a tray or a lane, and it organizes the spec, the examples and the labels around that assignment.

Pick-and-place and bin picking data are organized around the motion. This page puts the choice of destination at the center instead.

Visual inspection data covers looking, handling, rotating, checking and deciding about a part. Sorting often includes a check, such as inspecting a returned item or checking the condition of a damaged one, but it ends with the item placed in a destination rather than with a judgment about the item alone.

The task catalog lists sorting rows across many sectors, from retail and logistics to offices, hospitality, agriculture and healthcare. In a retail back of house, damaged-goods sorting means inspecting returned or damaged items, scanning them and sorting them into markdown or disposal bins, and a second row has the operator scan each damaged item, check its condition and drop it into the right returns bin.

Parcels and mail are sorted by route or lane: sorting letters and parcels by route into pigeonholes and trays at a sorting bench, scanning parcels and sorting them to lanes, and sorting outbound boxes by carrier or route after printing and applying labels. Offices sort incoming mail and documents into folders. Other rows sort stock onto labelled racks and bins, linens by type, laundry by colour, fabric and customer, and harvested crops by size and quality. The first table below lists each row with its description, environment and status.

The catalog places sorting in retail, warehouse, courier, office, hotel, laundry, farm, packing, care and childcare environments, each listed in the first table. Every task row carries a status that says how soon capture can begin.

The site defines those statuses plainly. Ready to deploy means contributors and site access are set up and capture starts once a spec is signed. Active collection means recording is running now, so a spec can join an existing capture stream. Available on request means the work is scoped and recruited to spec when a buyer asks for it. In the table below, 11 rows are Ready to deploy, 3 are Active collection and 12 are Available on request.

The warehouse robotics page describes its warehouse capture as taking place in working fulfilment space, on shift, rather than in a mock aisle built for the camera. The site does not say that any of the sorting capture takes place inside a particular customer facility, and this page does not claim it.

Recycling appears in the catalog as an environment: Recycling & Waste-Transfer Facility, with the examples sorting lines, baling and material recovery, described as a target environment for commercial egocentric collection and marked Ready to deploy. The catalog lists no separate task row under it, so the specific recycling sorting tasks, such as which materials go to which stream, are scoped in the spec. Two nearby rows do sort waste: e-waste disassembly and sorting, where devices are opened, boards and batteries removed and materials sorted into bins, and construction site cleanup, which includes sorting waste.

Rows named trash and recycling removal or sweep collect bags and carry them to bins. They are not sorting rows.

Returns are stated in several rows. Damaged-goods sorting in a retail back of house and returns processing in a warehouse both end with an item placed in a destination. A return processing row on a retail floor set inspects, re-tags and re-shelves a return, which is a different task, and it is not in the sorting table.

Parts sorting is not stated as a task row. The nearest rows sort materials into bins in e-waste disassembly, group backroom stock by category, and sort boxes and products by type. A buyer who needs parts sorted, for example by type or by the outcome of a check, describes the parts, the stations and the destinations in the spec, and the quote covers them before capture begins.

The written spec is the contract, so it is worth being specific about the parts of sorting the site leaves to the buyer. Name the items in scope at the level your policy needs, because the site does not label object identity by default, and name every destination: each bin, tray, cart, lane, rack or crate that an item can go to.

Name the rule that decides the destination, such as condition, size, route, type or colour, and say whether the operator applies it by looking, by scanning or by checking against a reference.

Name the start state and the end state, and what counts as a finished episode: a single item in its bin, a full tray or an emptied pile. Finally, name the capture mode: a head-mounted view of real work on site, or a staged set with handheld grippers for demonstrations that must map onto a robot end effector.

The catalog names two capture modes. In the first, a head-mounted phone records the first-person view of real work on site, and a multiview rig adds synchronized third-person angles when the spec asks for them. In the second, a staged working set is recorded by a multiview rig while the operator performs the task with handheld grippers, so demonstrations map onto a robot end effector.

A head-mounted camera keeps the item, the hand and the destinations in frame from the viewpoint a sorting policy will act from, with a wrist RGB stream alongside it for close-range detail. Depth, 3D hand pose, contact events and action segments are aligned to video on one hardware clock, synced under 2 ms. Distance-invalid pixels beyond the reliable depth range are flagged rather than interpolated.

The site does not state a camera count, resolution, frame rate or lighting setup specific to sorting, so those are agreed in the written spec rather than assumed. It also does not state how depth behaves on small, shiny or transparent items, or on mixed piles, so raise that in the spec and test it in a Spec Pilot.

Each episode follows the same episode schema used across Firsthand custom collections: ten streams synced under 2 ms, with head and wrist RGB, depth, 3D hand pose with per-joint visibility, contact and grasp events, and frame-accurate action segments. The warehouse and object rearrangement pages also list instance segmentation or masks among their outputs.

For sorting, hand pose tracks the operator through the pick-up, the check and the placement, contact events mark when fingers make and break contact with an item, and action segments mark the start and end of each step. Because the labels share one clock with the video and depth, a step boundary lines up with the exact frame where it happens.

Object labels use the terms your spec sets: the skill spec decides which object classes are in scope before capture, so labels follow the vocabulary your training stack expects rather than a generic taxonomy.

On the site stated terms for manipulation failures, yes: missed grasps, dropped items and mis-placed objects are captured and action-segmented alongside clean takes rather than discarded, because a policy that only sees clean takes has no model of recovery. That statement comes from the pick-and-place page, and it is what this page relies on for dropped and mis-placed items.

Several catalog rows build the decision into the task itself: checking the condition of a damaged item before choosing a returns bin, discarding damaged crops while grading, removing damaged fruit, and routing returned items to restock or disposal. The site does not state a per-episode sort-correctness or accuracy label, so a buyer who wants correct and incorrect placements tagged defines the tag in the spec.

Category labels, material-type labels, sort-correctness or accuracy labels, and object identity or pose labels are not stated anywhere on the site as standard labels. Neither is a per-episode destination label, a sort-rate figure or a count of items or categories.

Force, tactile and weight data are not stated either. One row sorts, weighs, packs and labels fresh strawberries using a bench scale, but the site does not describe a scale reading as a delivered stream. If your policy needs any of these, they are defined in the written spec as annotation depth, and the quote covers them before capture begins.

Every episode links to a signed contributor release through a documented consent chain, and the consent record is part of the delivery. Contributors are the people who perform the sorting tasks, so their release covers the footage captured of their hands and workspace.

Faces, plates and screens are detected and irreversibly blurred at ingest, and a reviewer confirms the result before anything ships. The anonymization record is delivered with the data. That matters in sorting settings, where coworkers, handheld scanner screens and workstation displays can enter the frame.

Parcel and shipping labels, mail, documents, folder labels, medication packaging and barcodes are not listed as blur categories on the site, so this page does not claim they are blurred. Mail and documents in particular can carry personal details. If text on items matters for your use, raise it in the spec so the handling is agreed before capture rather than assumed afterwards.

Delivery formats are RLDS, LeRobot and WebDataset, plus HDF5, zarr and Rerun .rrd. The first three are the training-loader formats; HDF5 and zarr carry arrays for custom pipelines, and Rerun .rrd gives a multi-stream synchronized playback file for review, which suits visually checking a sorting episode before writing a loader.

Converter source is readable, so you can retarget the schema to your own loader, and the episode schema page lists the fields.

You describe the items, destinations, sorting rules and stations your policy needs. We turn that into a written spec covering tasks, environments, capture tier, annotation depth and accept criteria, then quote reach, timeline and price against that document before any capture begins.

The collection runs through the same seven-stage end-to-end custom collection pipeline as every Firsthand program: contributor recruiting, capture, per-batch QA, anonymization and licensed delivery.

Where coverage is already deep, the first validated batch lands in a median of 48 hours from signed spec. Thin environments typically take three to four weeks, because vetting operators there is the bottleneck, and we quote that timeline up front. The catalog status on each task row tells you which case you are in.

On-demand top-ups run against the same spec and accept criteria after each training run, so new sorting data stays consistent with what you already hold. A Spec Pilot, a small and fast first batch, can prove the spec and accept criteria before you scale volume.

Every custom collection ships under a perpetual, buyer-owned commercial license, and exclusivity is available so the data is never resold. The license is documented alongside the consent chain, so a policy trained on the data can go into a product.