Egocentric Pick and Place Data Collection

Pick and place is the primitive underneath most manipulation tasks a robot policy is trained on, whether it is called picking in a warehouse, facing on a retail shelf, plating in a kitchen, or a tool hand-off at a bench. A policy learns the reach-to-grasp-to-release motion from the viewpoint it will actually act from, which incidental third-person footage was never captured to show.

Capturing it egocentrically, against a named skill list, means the pick-up and release moments are labeled against the same clock as the video and depth, rather than inferred after the fact from a single frame.

Pick and place is captured wherever it already appears across the catalog: single-item pick from bin to tote and carton break-down in warehouse picking, facing and restocking on a retail shelf, food prep handling in a kitchen, and tool hand-off between hands in tool use and assembly work.

A custom pick and place spec is scoped from that same task vocabulary — you describe the objects, start and goal locations, and setting your policy needs, and the spec is written to cover them rather than a generic "pick and place" category.

A head-mounted camera keeps the reach, grasp, transport, and place motion in frame from the viewpoint a manipulation policy will actually see from, with a wrist-mounted camera available alongside it for close-range grasp detail when that matters more than the full-arm reach.

Depth, 3D hand pose, contact events, and action segments are aligned to video on one hardware clock, synced under 2 ms. Distance-invalid pixels beyond the depth sensor’s reliable range are flagged rather than interpolated.

Episodes carry 3D hand pose tracked through reach, grasp, transport, and release with per-joint visibility flags, contact and grasp events at pick-up and release, and frame-accurate action segments marking the start and end of each phase.

Each episode is structured with the same episode schema used across Firsthand custom collections, so pick and place labels line up with the same fields a buyer’s loader already expects from other Firsthand datasets.

Missed grasps, dropped items, and mis-placed objects are captured and action-segmented alongside clean takes rather than discarded, because a policy that only sees a successful pick has no model of recovery. Warehouse picking in particular is captured on shift, where mis-picks and dropped totes occur as real failure cases.

Failure and recovery episodes are labeled against the same accept criteria as clean takes, so they count toward validated hours rather than being treated as waste.

You describe the objects, start and goal locations, and setting your policy needs to pick and place across. We turn it into a written spec covering tasks, environments, capture tier, annotation depth, and accept criteria, then quote reach, timeline, and price against that document before any capture begins.

The collection runs through the same seven-stage end-to-end custom collection pipeline as every Firsthand program: contributor recruiting, capture, per-batch QA, anonymization, and licensed delivery. Rejected items are recaptured at our cost.

Where coverage is already deep, the first validated batch lands in a median of 48 hours from signed spec. Thin environments typically take three to four weeks, because vetting operators there is the bottleneck, and we quote that timeline up front.

On-demand top-ups run against the same spec and accept criteria after each training run, so new pick and place data stays consistent with what you already hold. A Spec Pilot — a small, fast first batch ��� can prove the spec before you scale volume.

Every custom collection ships under a perpetual, buyer-owned commercial license, and exclusivity is available so the data is never resold. Each episode links to a signed contributor release through a documented consent chain.

Faces, plates, and screens are detected and irreversibly blurred at ingest, and a reviewer confirms the result before anything ships. Delivery formats are RLDS, LeRobot, and WebDataset, plus HDF5, zarr, and Rerun .rrd, with the anonymization record included.