Egocentric Bin Picking Video Data for Robot Training

Picking from a bin is one of the most common jobs in a fulfilment flow, and one of the hardest to show a policy well. A person reaches into a container, finds the item they want among others, closes a hand on it, lifts it clear and drops it into a tote. The site lists this as a named skill in its warehouse and logistics collections: single-item pick from bin to tote.

A first-person camera records that sequence from the viewpoint a robot would act from. The operator looks into the bin, chooses an item and watches the hand make contact, so the video shows not only what was done but what could be seen when it was done. Hand pose and contact events record the handling, and action segments mark where each step starts and ends. A manipulation policy can learn from this evidence without any robot being involved in the capture.

It helps to be exact about what the data is. It records a person picking: the views, the handling and the order of steps. The site does not state object pose, grasp-point, SKU identity, clutter-level, suction or pick-success labels, so this page does not promise them. A buyer who needs any of them defines them in the written spec before capture begins.

Warehouse robotics training data covers the whole fulfilment floor: pick, scan, pack and pallet-build across pick aisle, pack bench, pallet build and returns desk. Bin picking is one skill inside that list, and this page organizes the spec, the examples and the labels around that skill.

Egocentric pick-and-place data covers the general primitive of reach, grasp, transport and place wherever it appears, in warehouse picking, retail shelf work, kitchen tasks and tool hand-off. Grasping data focuses on the moment of the grasp itself. This page takes the bin and the tote as the setting: the item starts among other items in a container and ends in another one.

Packing and kitting is the reverse direction, putting items into a pack or kit. Logistics and fulfilment data is organized station by station, from receiving to loading. The related-pages table near the end sets each neighbour side by side.

The task catalog lists picking work in a warehouse and distribution setting: order picking, picking and packing customer orders with a handheld scanner, picking, scanning, packing and labelling orders, general warehousing and logistics work with scanners and totes, and each-pick to tote, described as picking individual items to a tote. A retail floor row, pick and pack order, uses the same motion at a pack bench.

It also lists tote and bin handling rows: tote induction (induct totes onto a conveyor), stacking empty totes (nest and stack empty totes) and, at a mock security checkpoint, bin stacking and return, loading screening trays (place bags and loose items into bins) and retrieving a flagged item from a bin. The first table below lists every row with its description, environment and status.

The site does not use the word singulation. Its wording for the core task is single-item pick from bin to tote, and this page uses that wording rather than a definition of its own.

The catalog places these tasks in four environment types: a warehouse and distribution center, a warehouse zone with conveyor, racking, pallets and a pack bench, a retail floor set, and a mock security checkpoint. Every task row carries a status that says how soon capture can begin.

The site defines those statuses plainly. Ready to deploy means contributors and site access are set up and capture starts once a spec is signed. Active collection means recording is running now, so a spec can join an existing capture stream. Available on request means the work is scoped and recruited to spec when a buyer asks for it. In the current catalog, the four warehouse and distribution rows are Ready to deploy, and the other seven are Available on request.

Warehouse capture is described as taking place in working fulfilment space, on shift, rather than in a mock aisle built for the camera. The site does not say that any of this capture takes place inside a particular customer facility, and this page does not claim it. The checkpoint rows are named mock in the catalog, so they are staged sets, not live ones.

The written spec is the contract, so it is worth being specific about the parts of bin picking the site leaves to the buyer. Name the source container and the destination, such as a bin, a tote or a carton, and the items in scope at the level your policy needs, because the site does not label item identity or SKU by default.

Name the picking context: a pick path walked with a handheld scanner, a single each-pick to a tote, or a pick and pack of a whole order. Name the tools around the pick, which the catalog mentions as scanners, picking carts, a tape dispenser, totes and handheld grippers.

Name the start state and the end state, and what counts as a finished episode. A single item in a tote and a completed order are different goals, so the accept criteria should say which one counts. Finally, name the capture mode: a head-mounted view of real work on site, or a staged set with handheld grippers for demonstrations that must map onto a robot end effector.

The catalog names two capture modes. In the first, a head-mounted phone records the first-person view of real work on site, and a multiview rig adds synchronized third-person angles when the spec asks for them. In the second, a staged working set is recorded by a multiview rig while the operator performs the task with handheld grippers, so demonstrations map onto a robot end effector.

A head-mounted camera keeps the pick, scan and place motion in frame from the viewpoint a warehouse policy will actually see from, with a wrist RGB stream alongside it for close-range detail. Depth, 3D hand pose, contact events and action segments are aligned to video on one hardware clock, synced under 2 ms. Pick aisles have long sight-lines, and distance-invalid pixels beyond the reliable depth range are flagged rather than interpolated.

The site does not state a camera count, resolution, frame rate or lighting setup specific to bin picking, so those are agreed in the written spec rather than assumed. It also does not state how depth behaves inside a deep bin or on shiny or transparent packaging, so raise that in the spec and test it in a Spec Pilot.

Each episode follows the same episode schema used across Firsthand custom collections: ten streams synced under 2 ms, with head and wrist RGB, depth, 3D hand pose with per-joint visibility, contact and grasp events, and frame-accurate action segments. The warehouse page also lists instance segmentation among its action-structure outputs, alongside action segments and grasp and contact events.

For bin picking, hand pose tracks the operator through the reach, the grasp and the release, contact events mark when fingers make and break contact with an item, and action segments mark the start and end of each step. Because the labels share one clock with the video and depth, a step boundary lines up with the exact frame where it happens.

Object labels use the terms your spec sets: the skill spec decides which object classes are in scope before capture, so labels follow the vocabulary your training stack expects rather than a generic taxonomy.

Yes, on the site stated terms. Missed grasps, dropped items and mis-placed objects are captured and action-segmented alongside clean takes rather than discarded, because a policy that only sees successful picks has no model of recovery. Warehouse picking in particular is captured on shift, where mis-picks and dropped totes occur as real failure cases.

The warehouse picking dataset page lists the conditions its fixed corpus covers, including Cluttered at 66 percent and Failure cases at 19 percent, described as mis-picks, dropped totes and jams. Those figures describe that fixed dataset, not a promise for a custom collection. The site does not state a per-episode clutter-level or pick-success label, so a buyer who wants failures tagged defines the tag in the spec.

Object pose, grasp-point, SKU identity, clutter-level, suction and pick-success labels are not stated anywhere on the site as standard labels. Package type and item-identity labels are not stated either, and neither is any separation or one-at-a-time label, since the site does not use the term singulation.

The capture modes include handheld grippers, but the site does not describe suction or a suction signal, and it does not state force or tactile data for warehouse collections. If your policy needs any of these, they are defined in the written spec as annotation depth, and the quote covers them before capture begins. The page does not promise them by default.

Every episode links to a signed contributor release through a documented consent chain, and the consent record is part of the delivery. Contributors are the people who perform the picking tasks, so their release covers the footage captured of their hands and workspace.

Faces, plates and screens are detected and irreversibly blurred at ingest, and a reviewer confirms the result before anything ships. The anonymization record is delivered with the data, so a buyer can see what was blurred and when. That matters in a working fulfilment space, where coworkers, handheld scanner screens and workstation displays can enter the frame.

Tote labels, carton markings, shipping labels and barcodes are not listed as blur categories on the site, so this page does not claim they are blurred. If label text in frame matters for your use, raise it in the spec so the handling is agreed before capture rather than assumed afterwards.

Delivery formats are RLDS, LeRobot and WebDataset, plus HDF5, zarr and Rerun .rrd. The first three are the training-loader formats; HDF5 and zarr carry arrays for custom pipelines, and Rerun .rrd gives a multi-stream synchronized playback file for review, which suits visually checking a picking episode before writing a loader.

Converter source is readable, so you can retarget the schema to your own loader, and the episode schema page lists the fields. Bin-picking episodes use the same schema as every other Firsthand collection, so a loader that already reads another Firsthand dataset reads a picking dataset without changes.

You describe the bins, totes, items, stations and end states your policy needs. We turn that into a written spec covering tasks, environments, capture tier, annotation depth and accept criteria, then quote reach, timeline and price against that document before any capture begins.

The collection runs through the same seven-stage end-to-end custom collection pipeline as every Firsthand program: contributor recruiting, capture, per-batch QA, anonymization and licensed delivery. Quality is measured against signed accept criteria, billing is per validated hour, a reject log ships with every batch, and rejected items are recaptured at our cost.

Where coverage is already deep, the first validated batch lands in a median of 48 hours from signed spec. Thin environments typically take three to four weeks, because vetting operators there is the bottleneck, and we quote that timeline up front. The catalog status on each task row tells you which case you are in.

On-demand top-ups run against the same spec and accept criteria after each training run, so new picking data stays consistent with what you already hold. A Spec Pilot, a small and fast first batch, can prove the spec and accept criteria before you scale volume.

Every custom collection ships under a perpetual, buyer-owned commercial license, and exclusivity is available so the data is never resold. The license is documented alongside the consent chain, so a policy trained on the data can go into a product.