Guide
Training Data for Household Robots
Short answer
Household robot training data needs long-horizon episodes across a whole home, not single tabletop grasps: laundry, tidying, and cleaning tasks with real navigation between rooms, deformable objects, and clutter. Firsthand ships this as episode-structured captures with per-frame instance masks, 3D hand pose, and labelled task boundaries across living spaces, bedrooms, bathrooms, and utility rooms.
Last reviewed:
Why is household robot data different from tabletop manipulation data?
A tabletop manipulation task is short and localized: reach, grasp, place, done, all within camera range of a fixed workspace. A household task is long-horizon and spans a whole home — a laundry cycle means walking to a hamper, carrying it to a machine, loading it, and later folding the output somewhere else entirely. The data has to capture that full arc, not just the grasp in the middle of it.
Household environments also introduce deformable objects (clothing, bedding, bags) and clutter that a controlled tabletop scene does not, which changes what the annotation has to carry.
What does household training data need to cover?
- Fold and sort laundry
- Deformable-object handling, bimanual coordination
- Load, transfer, and unload a washer
- Long-horizon sequencing across two locations
- Surface wipe and clutter reset
- Object recognition amid visual clutter
- Vacuum with obstacle negotiation
- Whole-room navigation, dynamic obstacle avoidance
- Bed-making, bimanual sheet handling
- Large deformable objects, two-handed coordination
- Bin change and bag tie
- Fine manipulation combined with navigation
Household task categories and what each one exercises.
These categories span multiple rooms — living space, bedroom, bathroom, and utility room — because a household robot has to operate across a whole home, not one fixed cell.
What annotation does household data need that tabletop data does not?
Deformable objects like clothing and bedding do not track well with propagated masks, so household captures ship per-frame instance masks rather than the frame-to-frame propagation that works for rigid objects. 3D hand pose and action segments with labelled task boundaries still apply, the same as any other capture, but the deformable-object masking is the added annotation cost specific to this environment.
How is a household robot collection specced?
Through a skill spec naming the task list (fold laundry, wipe a surface, vacuum with obstacle negotiation, and so on), the rooms it needs to span, and the participant and lighting mix — the same mechanism used for any custom collection, applied to a household task list instead of a tabletop or bimanual one.
FAQ
Training Data for Household Robots, answered.
01What makes household robot training data different from tabletop manipulation data?
Household tasks are long-horizon and span a whole home — walking between rooms, handling deformable objects like laundry and bedding, and navigating around clutter — rather than a single localized grasp on a fixed tabletop workspace.
02Why do deformable objects need different annotation?
Clothing and bedding do not track well with masks propagated frame-to-frame the way rigid objects do, so household captures ship per-frame instance masks instead, which is a higher annotation cost per hour.
03What rooms does household robot training data need to cover?
Living space, bedroom, bathroom, and utility room at minimum, since household tasks like laundry cycles and cleaning routines span multiple rooms rather than staying in one fixed location.
04Does household robot data still need 3D hand pose and action segments?
Yes — the same 3D hand pose, action segment, and task-boundary labelling used across every Firsthand capture applies to household episodes; the deformable-object instance masking is additional to that, not a replacement for it.
05How is a custom household robot dataset specified?
Through a skill spec naming the task list, the rooms it needs to span, and the participant and lighting mix — the same mechanism used to spec any custom collection, adapted to a household task list.
Check it against the sample pack.
40 episodes across 4 environments, delivered in the exact schema these guides describe.