Laundry Folding Video Data for Robot Training

Laundry folding video data for robot training is first-person video of a person folding, sorting or otherwise handling laundry, recorded from their own point of view with synchronized sensor streams alongside. The camera sits on the head, close to what a robot’s onboard camera would see over a folding surface, so the frames show what the hands are doing to the fabric rather than a distant view of someone standing in a room. The same idea sits behind any egocentric laundry video, applied here as a custom garment manipulation collection scoped to your tasks.

On this site the data is human capture: a paid, consented contributor wears the rig and performs a laundry task from a written skill spec. It is not robot or teleoperation data. This page is narrower than deformable-object data in general, which covers laundry alongside bedding and bags, and than household pretraining, which spans whole homes.

NVIDIA’s GR00T N1 paper describes a data pyramid with web data and human video at the base, synthetic data in the middle and real robot trajectories at the top. First-person human video of laundry handling sits in that base layer. It is far more abundant than robot data, and a person can move through many homes without the access and setup problems of placing an early-stage robot in each one. That is the practical case for cloth folding training data from people: the demonstrations come from the homes where a household robot will eventually work.

Laundry folding is also a hard benchmark for long, dexterous behavior. Physical Intelligence’s pi-zero paper describes a laundry folding task in which clothing items start crumpled in a bin, and the robot must take an item out, fold it and stack it. The paper notes that the policy has to handle clothing items that can start in any configuration, describes its multi-stage tasks, laundry folding among them, as taking between 5 and 20 minutes, and reports that such tasks needed fine-tuning on high-quality data.

Google DeepMind’s ALOHA Unleashed paper adds that its shirt-hanging task is long-horizon, involves a deformable object, and needs several dexterous bimanual behaviors. Both are robot teleoperation studies, not first-person human video, and Firsthand has no relationship with their authors. They show why the data has to cover many starting states and many steps. A custom collection lets you aim at the garments, homes and variation your model is missing, and because collection is on demand, a focused brief is a normal starting point.

Laundry stacks several capture problems that a rigid pick-and-place task does not have, and the first table on this page maps each one to the stream or label this site states for it.

First, the fabric itself. A garment has no single shape: it folds, drapes and bunches as it is handled, and part of it is always underneath another part. Clothing does not track well with masks propagated frame to frame, which is why the household datasets here ship per-frame instance masks, at the highest annotation cost per hour in the catalog.

Second, occlusion by the hands. Fabric covers fingers, and the folding hand often covers the very edge it is working on. 3D hand pose is delivered for both hands, 21 joints per hand, with a per-joint visibility flag, so an occluded finger is marked rather than hallucinated. Contact events mark when the hand meets the item, two wrist cameras give a closer second view of the hands than the head camera alone, and a batch is rejected if the manipulated object is out of frame for more than 15 percent of an episode.

Third, bimanual coordination. Folding depends on what the second hand is doing at the same instant: stabilizing, supporting, smoothing or handing off. A head-mounted camera keeps both hands in frame from the viewpoint a two-handed policy will see, and hand pose and contact events are recorded for each hand on the same clock as the video and depth.

Fourth, long multi-step sequences. A laundry cycle is described on this site as spanning a hamper, a hallway, a machine and a folding surface, and an early slip changes everything after it. That is why a laundry spec writes the steps and their order down, why each step becomes an action segment, and why failure and recovery cases can be required. Variety is the fifth problem: the site lists no garment taxonomy, so which garments a collection covers is a skill-spec decision.

A laundry collection uses the same capture stack as every Firsthand program. The instrumented tier is a head-mounted stereo rig that records first-person RGB, with a metric depth stream recovered from the calibrated stereo rig rather than from a standalone depth sensor, and 6-DoF IMU data at 200 Hz from head- and wrist-mounted sensors. The wider stack is ten streams: a 4K head camera, two wrist cameras, depth, IMU, 3D hand and body pose, instance segmentation and action labels, hardware-aligned under 2 ms.

Every stream is hardware-triggered onto one PTP clock domain, so depth, motion and hand pose line up with the same video frame rather than being approximated afterwards. The rig is calibrated twice per session, and an episode is accepted only if RMS reprojection stays under 0.28 px.

Each attempt is delivered as one episode, with action segments, 3D hand pose with visibility flags, contact events, objects and per-frame instance masks. Force and tactile readings are not among the streams listed, and no robot or teleoperation data is captured.

You describe the laundry task your pipeline needs covered, and that becomes a written skill spec. For laundry, the spec names the task and its sub-steps, their order, the setting, and the items involved. It defines where an attempt starts and ends and what counts as a completed attempt, such as a finished fold or a stacked pile. It also sets the capture tier, annotation depth and accept criteria, and states how much failure and recovery coverage to include alongside clean runs. Our skill spec guide walks through how to write one.

The laundry tasks this site already documents are folding and sorting laundry, and loading, transferring and unloading a washer. Bed-making with bimanual sheet handling and bin change and bag tie sit next to them in the deformable-object catalog. The site does not list other garments, towels or other soft items as separate tasks, and this page does not add them. A garment list, or a fold style beyond the sub-steps you write down, is a skill-spec question to settle in a Spec Pilot rather than an assumption.

Four spec choices matter most. Name the start state, because a garment that begins flat and one that begins crumpled in a basket are different tasks. Write the fold style as sub-steps and a completion rule, because there is no fold-quality label to define it afterwards. Name the homes and rooms, since a laundry cycle crosses several. And name the participant and lighting mix. How many homes, contributors, garments and hours a program covers is set per program in the skill spec.

Action segments divide an episode into start- and end-timestamped spans, one per sub-step, each carrying start_ns, end_ns, verb and noun. For laundry, a segment might be one fold, one smoothing pass or one placement on the stack, as written in the skill spec. Hand pose is delivered as hand_joints, a per-frame array for two hands with 21 joints each in metres in the head-camera frame, and hand_visible for the per-joint visibility flag. The other standard fields are contact_events, objects, and instance_masks, stored at the depth resolution of 848×480 so they line up with the depth stream.

Validation runs against signed accept criteria written into the spec before capture. Each batch goes through per-batch QA, and a reject log ships with it. Episodes are machine-checked against stated reject criteria: a sync breach beyond the ceiling for more than three consecutive frames, calibration drift beyond tolerance, the manipulated object out of frame for more than 15 percent of the episode, an undocumented bystander, or a mismatch against the named skill spec. A human reviewer then checks each episode against the task spec.

Only footage that clears the criteria is billed as a validated hour, and rejected items are recaptured at our cost. Many teams start with a Spec Pilot, a small, fast first batch that proves the spec and the accept criteria on real laundry footage before volume scales.

Fabric slips, bunches and needs a second attempt, and a policy trained only on clean runs has no model of what to do next. Failure and recovery cases are part of the same labeled structure as a clean attempt, and a written spec can require them in the batch. Our failure and recovery page covers this in more depth.

The robot-learning literature points the same way. The ALOHA Unleashed paper reports recovery and retry behavior in its shirt tasks, notes that some suboptimal data may be necessary because it contains such behavior, and reports that its shirt policy usually failed from starting configurations, such as a face-down shirt, that were absent from training. That is robot data, not Firsthand data, but it is why a laundry spec should name the starting states and slips it needs.

Laundry capture happens inside lived-in homes, including bedrooms, bathrooms and utility rooms. Every contributor signs a written release before capture, and each delivered episode links to that record through consent artefact IDs in the batch data card. The consent chain has six checkpoints, and a break at any one discards the footage.

At ingest, automated detection finds faces, plates and screens and blurs them irreversibly, and speech is scrubbed of names, addresses and card numbers while ambient audio is kept. A human reviewer confirms that no un-consented person is identifiable in any frame. Bathroom capture excludes any episode with a person in frame other than the consenting operator. If a person cannot be de-identified, the footage is discarded. The anonymization record is delivered with the data. None of this is legal advice, and the executed agreement controls.

Where this site does not state something, this page says "not stated" rather than filling the gap. The delivered-versus-not-stated table below separates the two. There is no listed cloth-state, garment-pose or fold-quality label, no garment taxonomy, no force or tactile stream, and no fold-success score. Nothing here claims a training outcome.

The tradeoff to plan for is the embodiment gap between a human hand and a robot end-effector, which retargeting or pairing with robot data has to close. Counts of homes, contributors, garments and hours are set in your skill spec.

Teleoperated robot demonstrations, as used in the pi-zero and ALOHA Unleashed papers, record what a specific robot did, so they match that robot’s embodiment exactly. Custom first-person human capture records what a person did, in a home, with no robot in the loop, but has to be retargeted to a robot. The two are complements: the pi-zero paper itself describes broad pretraining followed by fine-tuning on narrower, higher-quality data.

The comparison below keeps the third-party column to what those papers state about their own data. Whatever dataset you use, read its license and consent terms before training on it.

Delivery is in the format your stack already reads: RLDS, LeRobot, WebDataset, HDF5, zarr or Rerun .rrd, or raw streams with metadata. Readable converter source ships alongside the data so the schema can be retargeted to your own loader. Collections are billed per validated hour.

The license is perpetual, irrevocable and worldwide, with the data buyer-owned and non-exclusive by default. Training, evaluating, fine-tuning and deploying models, including commercial ones, is permitted, and exclusivity is available as a separate term. The executed agreement controls.

From signed spec to first validated batch, the timeline is a median of 48 hours where coverage is deep and three to four weeks where it is thin and contributors need to be recruited first. After the first batch, on-demand top-ups run against the same accept criteria, so you can add more garments, homes or variation without rewriting the spec.

Describe the laundry tasks, the garments and starting states, the sub-steps of each fold and their order, the homes and rooms, and the failure or variation cases your pipeline is missing. That becomes a written skill spec, quoted for reach, timeline and price before any capture begins, and most programs run a Spec Pilot first.