Robot learning
Manipulation policies learn from hands, not captions.
- Source
- Human demonstration
- Signals
- Hand pose · contact · depth
- Formats
- RLDS · LeRobot
- License
- Buyer-owned
What it is
Demonstration, captured where contact happens.
Robot manipulation data is the record a policy learns to handle objects from — the first-person view plus the hand pose, contact events, and depth that describe how the interaction actually unfolded.
A caption can say "pick up the mug." A policy needs to know where the handle was, how the hand approached, and the instant the fingers closed. Those signals only exist if you capture them at the moment of contact, from a viewpoint that matches where the policy will run — which is why we build manipulation data from egocentric human demonstration rather than scraped clips.
Capturing people instead of a single teleoperated robot lets us cover far more objects, grasps, and environments, far faster. The hands-in-frame view transfers directly to a wrist- or head-mounted policy, and every episode carries the consent record and reject log we attach to all Firsthand data.
What we capture
The signals a grasp is made of.
- First-person grasp view
- Head-mounted RGB with the hands and manipulated object in frame through the whole reach-grasp-release cycle.
- 3D hand pose
- Per-frame hand keypoints with visibility flags, so occluded fingers are marked, not guessed.
- Contact events
- When and where fingers make and break contact — the signal manipulation policies actually condition on.
- Metric depth
- Aligned depth for object distance, approach vectors, and collision reasoning.
- Wrist camera
- Optional close-range wrist view of the interaction for fine manipulation.
- Object & scene metadata
- The objects, their arrangement, and the environment sampled to your coverage matrix.
Hand-pose accuracy and sync are measured, not assumed — see the numbers on benchmarks and the method on capture methodology.
Where teams use it
From imitation learning to VLA policies.
- Imitation learning and behavior cloning
- Vision-language-action (VLA) policies
- Grasp and dexterous-manipulation models
- Teleoperation-to-policy bootstrapping
- Reward and affordance learning from contact
- Benchmarking manipulation generalization
FAQ
Questions teams ask first.
- What is robot manipulation data?
- Robot manipulation data is the demonstration record a policy learns to grasp and handle objects from — the first-person view of a task plus the hand pose, contact events, and depth that describe how the interaction happened. It can come from teleoperated robots or, as we capture it, from real people performing the task, which scales across environments far faster than a robot fleet.
- Do you use human demonstration or robot teleoperation?
- Human demonstration by default. Capturing people doing the task on our egocentric rig lets us cover many more environments, objects, and edge cases than a fixed robot setup, and the hands-in-frame viewpoint transfers directly to a wrist- or head-mounted policy. When a program specifically needs teleoperated robot trajectories, we design that explicitly rather than blurring the two.
- How do you capture contact and hand pose reliably?
- On a synchronized multi-sensor rig: 3D hand keypoints are tracked through the sequence with visibility flags so occluded fingers are marked rather than hallucinated, and contact events are aligned to the same clock as the video and depth. The median cross-stream offset is verified under two milliseconds and reported with the batch.
- What formats do you deliver manipulation data in?
- RLDS and LeRobot for policy training, plus WebDataset, HDF5, zarr, and Rerun .rrd. Every episode carries int64 nanosecond timestamps per stream and calibration, and the converters ship as readable source so you can retarget to your own loader.
- Can you collect manipulation data for specific objects or tasks?
- Yes. We collect against a named spec — the objects, grasps, environments, and failure cases you need — sourced from a vetted global crowd across 150+ countries and 500+ languages. If your task is not in an existing environment, our custom-collection service builds the program around it.
Scope manipulation data against your objects.
Tell us the grasps, objects, and environments your policy has to generalize across. You will get a scoped estimate — signals, reach, timeline, and price — before any commitment.