Egocentric Tool Use Data Collection

A policy that hands off between tools, selects the right bit, or recovers from a slipped grip needs to have seen that motion from the viewpoint it will act from. Public benchmark datasets for bimanual precision, such as EgoDex, have genuinely strong hand pose, but their licenses forbid commercial training use and forbid derivatives, so they cannot legally reach production weights.

Bench tool use is also the cleanest environment to run this kind of capture in, because the operator controls the whole scene: lighting, clutter, and tool layout can be varied deliberately rather than relying on incidental footage that was never captured against a named skill list.

Tool use and assembly coverage on this site spans home workshop, maker bench, and small production cell environments, with a named skill list: driver and bit selection and drive-to-seat, clamp, align, and fasten a joint, measure and mark to tolerance, press-fit and interference assembly, wire strip, crimp, and terminate, and tool hand-off between hands.

A custom tool use spec is scoped from that same task vocabulary — you describe the tools and tasks your policy needs, and the spec is written to cover them rather than a generic "tool use" category.

A head-mounted camera keeps the tool-handling motion in frame from the viewpoint a manipulation policy will actually see from. Depth, 3D hand pose, contact events, and action segments are aligned to video on one hardware clock, synced under 2 ms.

Small tools and fasteners below the depth sensor floor are flagged rather than interpolated, while hand pose remains valid at that range, so the delivered data states exactly where metric depth stops being trustworthy.

Episodes carry 3D hand pose tracked through reach, grasp, and tool manipulation with per-joint visibility flags, contact and grasp events at the point of tool engagement, and frame-accurate action segments marking task boundaries such as pick-up, engage, and release.

Each episode is structured with the same episode schema used across Firsthand custom collections, so tool use labels line up with the same fields a buyer’s loader already expects from other Firsthand datasets.

Cross-threads, dropped fasteners or tools, slipped grips, and misalignment are captured and action-segmented alongside clean takes rather than discarded, because a policy that only sees successful tool use has no model of recovery.

Failure and recovery episodes are labeled against the same accept criteria as clean takes, so they count toward validated hours rather than being treated as waste.

You describe the tool use tasks your policy needs. We turn it into a written spec covering tasks, environments, capture tier, annotation depth, and accept criteria, then quote reach, timeline, and price against that document before any capture begins.

The collection runs through the same seven-stage end-to-end custom collection pipeline as every Firsthand program: contributor recruiting, capture, per-batch QA, anonymization, and licensed delivery. Rejected items are recaptured at our cost.

Where coverage is already deep, the first validated batch lands in a median of 48 hours from signed spec. Thin environments typically take three to four weeks, because vetting operators there is the bottleneck, and we quote that timeline up front.

On-demand top-ups run against the same spec and accept criteria after each training run, so new tool use data stays consistent with what you already hold. A Spec Pilot — a small, fast first batch — can prove the spec before you scale volume.

Every custom collection ships under a perpetual, buyer-owned commercial license, and exclusivity is available so the data is never resold. Each episode links to a signed contributor release through a documented consent chain.

Faces, plates, and screens are detected and irreversibly blurred at ingest, and a reviewer confirms the result before anything ships. Delivery formats are RLDS, LeRobot, and WebDataset, plus HDF5, zarr, and Rerun .rrd, with the anonymization record included.