Egocentric Deformable Object Manipulation Data

Egocentric deformable object manipulation data is first-person video of a person working with an item that changes shape as it is handled, recorded from their own point of view with synchronized sensor streams alongside. On this site the deformable items are clothing, bedding and bags, and the tasks are the laundry, bed-making and bag-handling routines listed below.

This page covers buying a custom collection of those demonstrations. It is narrower than a general dexterous-manipulation collection and than household pretraining at scale: the subject here is the deformable task itself, how it is specified, and how it is labeled so a training pipeline can use it.

A rigid object keeps one shape, so its position and orientation describe it. A towel, a sheet or a bag has no single shape: it folds, drapes and bunches as it is handled, and part of it is usually hidden under the hand. Three consequences show up in the data, and each is something this site already addresses rather than something we add here.

First, tracking. Clothing, bedding and bags do not track well with masks propagated frame to frame, which is why the household datasets here ship per-frame instance masks, at the highest annotation cost per hour in the catalog. Second, occlusion: fingers disappear under fabric, so hand pose needs a per-joint visibility flag. Third, coordination: many soft-object tasks, from opening a jar to folding fabric, depend on what the second hand is doing at the same instant.

We do not capture robot or teleoperation data, and nothing here claims a training outcome. The tradeoff to plan for is the embodiment gap between a human hand and a robot end-effector, which retargeting or pairing with robot data has to close. What you get is human first-person demonstrations in which the hands, the item and the step structure are all explicit.

The deformable tasks this site already lists are: fold and sort laundry; load, transfer and unload a washer; bed-making with bimanual sheet handling; and bin change and bag tie. A laundry cycle is described as spanning a hamper, a hallway, a machine and a folding surface, so a spec can cover the whole arc or one stage of it.

These are captured in home settings: living space, bedroom, bathroom, utility room, and stairs and hallway. The site does not list other garments, towels or other soft items as tasks, and this page does not add them. If your task is not on the list, treat it as a skill-spec question to settle in a Spec Pilot rather than an assumption.

You describe the task your pipeline needs covered and that becomes a written skill spec. For a deformable task it names the task and its sub-steps, their order, the setting, and the items involved. It also defines where an attempt starts and ends and what counts as a completed attempt, such as a finished fold or a made bed.

That definition lives in the spec and in the episode boundaries. It is not a separate cloth-state or garment-pose label, because none is listed among the delivered fields. The spec also sets the capture tier, annotation depth and the accept criteria, and states how much failure and recovery coverage to include alongside clean completions.

It is quoted for reach, timeline and price before any capture begins, and the accept criteria are signed into it, so acceptance is measured against what was agreed rather than judged afterwards. The skill spec guide linked below walks through how to write one.

Paid, consented contributors perform the real task against the spec while wearing the capture rig. The instrumented tier is a head-mounted stereo rig: first-person RGB, with a metric depth stream recovered from the calibrated stereo rig rather than from a standalone depth sensor, and 6-DoF IMU data at 200 Hz from head- and wrist-mounted sensors.

Every stream is hardware-triggered onto one PTP clock domain, with int64 nanosecond timestamps per stream, so depth, motion and hand pose are aligned to the same video frame rather than approximated afterwards. The rig is calibrated before capture and again after the session ends. Force and tactile readings are not among the streams listed here.

Collection is on demand: you do not need a standing program, and a focused brief is a normal starting point.

Each attempt is one episode. Inside it, action segments divide the task into start- and end-timestamped spans, one per sub-step, each carrying start_ns, end_ns, verb and noun. The other fields are the standard episode schema: hand_joints and hand_visible for 3D hand pose with a per-joint visibility flag, contact_events for the moment the hand meets the item, objects, and instance_masks.

For deformable items the instance masks matter most. The site states they are shipped per frame rather than propagated, and that instance masks are stored at the depth resolution, 848×480, so they line up with the synchronized depth stream. None of these are extra fields specific to deformable capture; the table below maps each label to its delivery field.

Soft items slip, bunch and need a second attempt, and a policy trained only on clean runs has no model of what to do next. Failure and recovery cases are part of the same labeled structure as a clean attempt, and a written spec can require them in the batch. Our failure and recovery page covers this in more depth.

Acceptance is measured against signed accept criteria written into the spec before capture begins. Each batch goes through per-batch QA, and a reject log ships with every batch. An episode is accepted only if RMS reprojection stays under 0.28 px and the drift is within tolerance; otherwise it goes to the reject log.

Only footage that clears the criteria is billed as a validated hour, and rejected items are recaptured at our cost. Many teams start with a Spec Pilot: a small, fast first batch that proves the spec and the accept criteria before you scale volume.

The buyer owns it. Every custom collection is delivered under a perpetual, non-exclusive commercial license by default, with exclusivity available, so the dataset can be trained on and productized without a research-only or unstated arrangement. Every episode is linked to a signed contributor release under a documented consent chain, and faces and screens in frame are detected and irreversibly blurred at ingest, with a reviewer confirming the result before anything ships. Clips that cannot be de-identified are discarded rather than delivered.

Delivery is in the format your stack already reads: RLDS, LeRobot, WebDataset, HDF5, zarr or Rerun .rrd, or raw streams with metadata. Readable converter source ships alongside the data so the schema can be retargeted to your own loader.

From signed spec to first validated batch, the timeline is a median of 48 hours where coverage is deep and three to four weeks where it is thin and contributors need to be recruited first. After the first batch, on-demand top-ups run against the same accept criteria, so you can add more attempts or more variation without rewriting the spec.

Describe the deformable task, its sub-steps and their order, the setting, and the failure or variation cases your pipeline is missing. That becomes a written skill spec, quoted for reach, timeline and price before capture begins, and most programs run a Spec Pilot first.