Household Robot Pretraining Data

Household robot pretraining data sits at the same base layer NVIDIA’s GR00T N1 describes in its data pyramid: web data and human video at the base because they are the most abundant, synthetic data in the middle, and real robot trajectories at the top because they are the scarcest and most expensive to collect. For a household robot specifically, that base layer is egocentric human video of home tasks — laundry, tidying, cleaning, and surface reset — captured from a viewpoint close to what a robot’s onboard camera would see inside a home.

That pyramid framing is why household pretraining leans on human capture rather than robot teleoperation logged inside real homes: a person wearing a capture rig can move through far more homes, rooms, and tasks per hour than a teleoperated robot arm can, and doing it with a person avoids the safety and access problems of running an early-stage robot through someone’s bathroom or kitchen before it is ready.

A household robot deployed at scale will operate inside homes it never saw during training — different layouts, different furniture, different clutter, different people. A model fine-tuned on one house learns that house; it does not learn to generalize to the next one. Pretraining-scale data is what gives the model a broad enough prior over home layouts, objects, and tasks that fine-tuning on any single deployment home becomes a narrowing step rather than the model’s entire education.

EgoScale documents the general shape of this recipe: broad human pretraining first, establishing a log-linear scaling law between the amount of human pretraining data and validation loss, followed by a smaller, aligned mid-training stage matched to the target embodiment, and then task-specific fine-tuning. On a 22-degree-of-freedom dexterous hand, EgoScale reports a 54% average improvement in success rate over training with no human pretraining at all — evidence that the broad pretraining layer, not just the narrow fine-tuning stage, is doing real work. The same logic applies to a household robot: the pretraining corpus has to be broad across homes before any one deployment’s fine-tuning data can specialize effectively.

Generalist’s own framing from GEN-0 reinforces the same point in different words: "data quality and diversity matters more than sheer volume." A household pretraining corpus that repeats the same living room a thousand times does not generalize any better than one that visits a hundred distinct homes once each — diversity of home, room, and object is the variable that matters, not raw hour count alone.

A household pretraining corpus needs the same long-horizon task categories the household-chores dataset already documents on this site — laundry, cleaning, and surface-reset tasks that exercise deformable-object handling, whole-room navigation, and bimanual coordination ��� but captured across many distinct homes rather than concentrated in a small set of recorded sites.

The same ten-stream capture stack used across every Firsthand collection applies to household pretraining: 4K head camera, two wrist cameras, depth, IMU, 3D hand and body pose, instance segmentation, and action labels, hardware-aligned under 2 ms. Because video alone under-specifies what happened in a task, hand pose, contact events, and action segments turn raw home footage into a signal a policy can actually learn a prior from.

The one addition specific to household environments is deformable-object masking. Clothing, bedding, and bags do not track well with masks propagated frame-to-frame the way rigid tabletop objects do, so household captures ship per-frame instance masks instead of the cheaper propagated masking that works for a rigid object on a workbench. That is the highest annotation cost per hour in the catalog, and it is what makes household footage usable for training rather than just watchable.

Real robot trajectories sit at the top of the GR00T N1 data pyramid because they are the scarcest and most expensive tier to collect, and that is especially true inside homes: running a teleoperated robot through a stranger’s kitchen or bathroom raises safety and access constraints that a person wearing a capture rig does not face. Egocentric human video gives a household foundation model the same first-person viewpoint and physically grounded motion a robot will act from, at a fraction of the hardware cost and without needing a robot on hand in every home visited.

That is also why groups pursuing pretraining scale, like Generalist’s GEN-0 program, describe accumulating hundreds of thousands of hours of real-world manipulation data and adding more than 10,000 new hours every week — a rate of coverage that is only realistic with human capture, not robot teleoperation logged one home at a time. Robot data still has a role: EgoScale’s own mid-training stage pairs roughly 50 hours of human video with about 4 hours of robot data across 344 tabletop tasks, a small, closely aligned dataset that comes after the broad human pretraining layer, not instead of it. The same ratio applies to a household program — broad human pretraining across many homes, then a smaller, closely aligned robot mid-training stage on the specific embodiment being deployed.

Diversity planning for a household pretraining corpus works the same way it does for any Firsthand collection: a written spec names the task list, the room types it needs to span, and the participant and lighting mix, rather than leaving coverage to whatever homes happen to be convenient. For household work specifically, that spec should name the room types explicitly �� living space, bedroom, bathroom, and utility room at minimum — since household tasks like laundry cycles and cleaning routines span multiple rooms rather than staying in one fixed location.

Object diversity matters as much as room diversity. A pretraining corpus concentrated in homes with similar furniture, similar laundry hampers, and similar clutter will generalize about as narrowly as one concentrated in a single home. Spanning distinct households — different layouts, different object sets, different lighting conditions — is what gives the pretraining layer a prior broad enough that a specific deployment home’s fine-tuning data becomes a narrowing step rather than the model’s entire education.

Participant diversity closes the loop: capturing the same tasks performed by different people, at different paces and with different habits, keeps a model from overfitting to one person’s particular way of folding a towel or loading a washer. None of these three — rooms, objects, people — substitutes for the other two; a spec that only varies one of them still leaves a gap the model will hit on its next unseen home.

Household capture happens inside real, lived-in spaces, which raises the bar above what a lab or workshop capture needs. Every custom collection ships under a documented consent chain, with each episode linked to a signed contributor release, and faces, plates, and screens are detected and irreversibly blurred at ingest, with a reviewer confirming the result before anything ships.

Bathroom capture specifically excludes any episode with a person in frame other than the consenting operator, a rule that reflects how much more privacy-sensitive in-home spaces are compared with a controlled tabletop or workshop environment. The anonymization record is delivered with the data, so a household pretraining program has a clear paper trail from contributor consent through anonymization to the license terms governing commercial training on the finished corpus.

Public egocentric corpora referenced across pretraining research — EgoScale, GEN-0, EgoVerse, Egocentric-10K — are captured broadly across everyday human behavior, not scoped specifically to household robot task distributions, room types, or the deformable-object masking a household program needs. Most also carry research-only or unstated licensing terms rather than a right to train a commercial household robot product on them.

A custom household pretraining collection closes that gap by capturing to a written spec across the specific homes, rooms, and tasks a model is missing, with hardware-synchronized streams and per-frame deformable-object masks throughout, under a license written for commercial training from the outset. It also closes the coverage gap public corpora cannot: a public dataset was captured for someone else’s task distribution, so it will always be missing exactly the home layouts or object types a given household model needs next.

You describe the home layouts, rooms, and tasks your pretraining corpus is missing. That becomes a written spec covering the task list, the room types it needs to span, the participant and lighting mix, capture tier, annotation depth, and accept criteria, quoted for reach, timeline, and price before any capture begins. Many programs start with a Spec Pilot, a small, fast first batch that proves the spec and accept criteria before scaling volume.

From there the collection runs through the same seven-stage end-to-end pipeline as every Firsthand program: contributor recruiting, capture across the specified homes, per-batch QA, anonymization, and licensed delivery, with rejected items recaptured at our cost and on-demand top-ups running against the same accept criteria afterward. A median 48-hour turnaround on a first batch means a household pretraining program can validate its spec — room coverage, object diversity, mask quality — quickly, before scaling the same pipeline to the volume the model needs.

RLDS, LeRobot, and WebDataset, plus HDF5, zarr, and Rerun .rrd, so a household pretraining corpus loads directly into the same pipelines used for mid-training and fine-tuning data. Converter source is readable, so you can retarget the schema to your own loader.

Start with the room types, home layouts, or tasks your current household corpus is weakest on rather than a total hour target. That gap becomes the written spec — task list, rooms, participant and lighting mix, capture tier, annotation depth, and accept criteria — quoted for reach, timeline, and price up front.

A Spec Pilot proves that spec on a small, fast first batch before any commitment to scale, so you can check per-frame mask quality, hand-pose fidelity, and format compatibility with your training pipeline before volume ramps. From there, the same seven-stage pipeline scales the corpus to the size the model needs, with on-demand top-ups available afterward as new gaps in home or room coverage show up during training.