Manipulation

Dexterous Manipulation Training Data

Fine-grained manipulation lives in the fingers, not the wrist. We capture it egocentrically, with 3D hand pose and contact events tracked through occlusion, against a spec you write.
Streams
Ten, synced under 2 ms
First batch
Median 48 h
Billing
Per validated hour
License
Buyer-owned
A contributor wearing a head-mounted camera beside a graphic of the first-person feed and annotation layers it produces.
Head-worn capture keeps fingers and object in frame through the whole approach, grasp, and adjustment cycle.

Requirements

What does dexterous manipulation data need?

01

Fingers tracked, not just the wrist

Fine manipulation regrasping, in-hand reorientation, precision pinches depends on finger-level pose, not a single wrist or gripper position.

02

Contact events at the fingertip

Contact events aligned to the same clock as video and depth mark exactly when and where each part of the hand makes and breaks contact.

03

Occlusion marked, not guessed

When the object or the hand itself occludes fingers, visibility flags mark it rather than letting a tracker hallucinate a pose.

04

Rights you can ship on

A perpetual, buyer-owned commercial license backed by a documented consent chain, so a policy trained on the data can go into a product.

Why it matters

Why is dexterous manipulation hard to capture well?

Most manipulation data defaults to a wrist-level pose or a parallel-gripper state because it is simpler to log. But fine manipulation, tool use, in-hand reorientation, precision pinches happens at the fingertip, and that signal is lost the moment capture drops to wrist level.

Capturing it requires 3D hand pose tracked through the full reach-grasp-adjust cycle, with visibility flags for the fingers an object or the hand itself occludes, not a wide shot that loses finger detail.

Capture setup

How is dexterous manipulation data captured egocentrically?

A head-mounted camera keeps the hand and manipulated object in frame from the viewpoint a manipulation policy will actually see from, through the whole reach-grasp-release cycle. An optional wrist camera adds close-range detail for fine manipulation.

3D hand pose is tracked with visibility flags, and contact events mark exactly when and where the hand makes and breaks contact, all aligned to depth and video on one clock.

The spec

How does a custom dexterous manipulation spec work?

You describe the manipulation tasks your policy needs. We turn it into a written spec covering tasks, environments, objects, capture tier, annotation depth, and accept criteria, then quote reach, timeline, and price against that document before any capture begins.

The collection runs through the same seven-stage end-to-end custom collection pipeline as every Firsthand program: contributor recruiting, capture, per-batch QA, anonymization, and licensed delivery. Rejected items are recaptured at our cost.

Fast · on demand

How fast does a first batch ship?

Where coverage is already deep, the first validated batch lands in a median of 48 hours from signed spec. Thin environments typically take three to four weeks, because vetting operators there is the bottleneck, and we quote that timeline up front.

On-demand top-ups run against the same spec and accept criteria after each training run, so new dexterous manipulation data stays consistent with what you already hold.

Rights

What do licensing and consent look like?

Every custom collection ships under a perpetual, buyer-owned commercial license, and exclusivity is available so the data is never resold. Each episode links to a signed contributor release through a documented consent chain.

Faces, plates, and screens are detected and irreversibly blurred at ingest, and a reviewer confirms the result before anything ships. The anonymization record is delivered with the data.

Dexterous manipulation need vs delivery

What does a dexterous manipulation policy get, need by need?

Dexterous manipulation training needs and what a Firsthand custom collection delivers for each.
Dexterous manipulation needWhat Firsthand captures
Hand and object in frameHead-mounted camera keeping the hand and object visible through the full task cycle
Finger-level pose3D hand pose tracked through the sequence, with visibility flags for occlusion
Contact detailContact events aligned to the same clock as video and depth
Fine manipulation detailOptional wrist camera for close-range detail on the hand
DepthSynchronized depth stream for scene and contact geometry
Action structureInstance segmentation, action segments, grasp and contact events
QualityMeasured against signed accept criteria; billed per validated hour; reject log with every batch
FormatsRLDS, LeRobot, WebDataset, plus HDF5, zarr, and Rerun .rrd

Every collection runs through the same seven-stage end-to-end custom collection pipeline.

FAQ

Dexterous manipulation training data, answered.

What is dexterous manipulation training data?
It is real-world, first-person demonstration data of fine-grained, multi-finger manipulation captured against a written spec: egocentric video with 3D hand pose, contact events, and depth. Firsthand handles contributors, capture, QA, consent, and delivery.
How is this different from general manipulation data?
General manipulation data can log a wrist or gripper position. Dexterous manipulation data tracks 3D hand pose through the reach-grasp-adjust cycle with visibility flags, so finger-level detail is captured rather than lost.
Does capture include contact events at the fingertip?
Yes. Each episode includes 3D hand pose and contact events aligned to the same clock as the video and depth streams.
How is custom dexterous manipulation data billed?
Per validated hour, quoted against the written spec before any capture. You pay only for data that clears the accept criteria, and rejected items are recaptured at our cost.
Which delivery formats are supported?
RLDS, LeRobot, and WebDataset, plus HDF5, zarr, and Rerun .rrd. Converter source is readable, so you can retarget the schema to your own loader.
Can we start with a small pilot?
Yes. A Spec Pilot is a small, fast first batch that proves the spec and accept criteria before you scale volume on demand.

Sources

Where are the delivery formats documented?

Request a custom dataset.

Tell us the dexterous manipulation tasks your policy needs to perform. We will quote reach, timeline, and price against a written spec, with a first validated batch in a median of 48 hours where coverage is deep.