Kitchen Task Pretraining Data for Robots

Kitchen task pretraining data sits at the same base layer NVIDIA’s GR00T N1 describes in its data pyramid: web data and human video at the base because they are the most abundant, synthetic data in the middle, and real robot trajectories at the top because they are the scarcest and most expensive to collect. For a robot that will manipulate objects in a kitchen, that base layer is egocentric human video of food prep, cleaning, and loading and unloading tasks, captured from a viewpoint close to what a robot’s onboard camera would see over a counter.

That pyramid framing is why kitchen pretraining leans on human capture rather than robot teleoperation logged in real kitchens: a person wearing a capture rig can move through far more kitchens, tasks, and contact events per hour than a teleoperated robot arm can, and doing it with a person avoids the safety and access problems of running an early-stage robot near knives, hot surfaces, and hot liquids before it is ready.

A kitchen concentrates more manipulation variety per square foot than almost any other environment we capture: knife work, pouring and decanting, opening jars and packaging, loading and unloading a dishwasher, stovetop transfers, and wiping down a surface, often within a single episode. That density of contact events and object handoffs makes kitchen data efficient to capture relative to the manipulation variety it yields, compared with environments where the same few actions repeat with less variation.

EgoScale documents the general shape of the recipe that kitchen pretraining data feeds into: broad human pretraining first, establishing a log-linear scaling law between the amount of human pretraining data and validation loss, followed by a smaller, aligned mid-training stage matched to the target embodiment, and then task-specific fine-tuning. On a 22-degree-of-freedom dexterous hand, EgoScale reports a 54% average improvement in success rate over training with no human pretraining at all — evidence that the broad pretraining layer, not just the narrow fine-tuning stage, is doing real work. Kitchens, with their density of grasps and tool changes, are a natural place to build that broad pretraining layer for dexterous manipulation.

Generalist’s own framing from GEN-0 reinforces the same point in different words: "data quality and diversity matters more than sheer volume." A kitchen pretraining corpus that repeats the same counter and the same three recipes a thousand times does not generalize any better than one that visits a hundred distinct kitchens once each — diversity of kitchen, task, and object is the variable that matters, not raw hour count alone.

A kitchen pretraining corpus needs the same task families the egocentric kitchen video dataset already documents on this site — knife work, pouring and decanting, opening jars and packaging, dishwasher loading and unloading, stovetop transfers, surface reset, and bimanual dough handling — but captured across many distinct kitchens and contributors rather than concentrated in a small set of recorded sites.

Scene diversity matters alongside task diversity: domestic kitchens, shared-house kitchens, and small commercial prep lines each present different counter layouts, lighting, and clutter patterns, and a spec that only covers one of them leaves a gap a model will hit the first time it sees a different kitchen type.

The same ten-stream capture stack used across every Firsthand collection applies to kitchen pretraining: 4K head camera, two wrist cameras, depth, IMU, 3D hand and body pose, instance segmentation, and action labels, hardware-aligned under 2 ms. Because video alone under-specifies what happened in a task, hand pose, contact events, and action segments turn raw kitchen footage into a signal a policy can actually learn a prior from.

Wet and reflective surfaces — a rinsed pan, a wet countertop, a steel sink — degrade active-stereo depth, so kitchen captures ship per-frame depth confidence maps alongside the depth stream, letting you mask low-confidence regions rather than guess. Steam events are labelled as an episode-level flag, since they can disrupt depth for several seconds at a time. Deformable objects such as dough or produce ship per-frame instance masks rather than the cheaper propagated masking that works for rigid tabletop objects. Data ships in RLDS, LeRobot, and WebDataset, plus HDF5, zarr, and Rerun .rrd, so a kitchen pretraining corpus loads directly into the same pipelines used for mid-training and fine-tuning data.

Diversity planning for a kitchen pretraining corpus works the same way it does for any Firsthand collection: a written spec names the task list, the kitchen and scene types it needs to span, and the participant and lighting mix, rather than leaving coverage to whatever kitchens happen to be convenient. That spec should name scene types explicitly — domestic kitchen, shared-house kitchen, and small commercial prep line at minimum — since each presents a different counter layout and clutter pattern.

Object diversity matters as much as scene diversity. A pretraining corpus concentrated in kitchens with similar cookware, similar produce, and similar clutter will generalize about as narrowly as one concentrated in a single kitchen. Lighting conditions matter too: daylight coverage fills in fastest, while low-light, evening capture tends to be the harder cell to fill, and a spec that does not name it explicitly risks shipping a corpus biased toward daytime cooking.

Participant diversity closes the loop: capturing the same tasks performed by different people, at different paces and with different habits — left- and right-handed knife work, different chopping styles, different pours — keeps a model from overfitting to one person’s particular technique. None of these three — kitchens, objects, people — substitutes for the other two; a spec that only varies one of them still leaves a gap the model will hit on its next unseen kitchen.

Kitchen capture happens inside real, lived-in spaces, which raises the bar above what a controlled workshop capture needs. Every custom collection ships under a documented consent chain, with each episode linked to a signed contributor release, and faces, plates, and screens are detected and irreversibly blurred at ingest, with a reviewer confirming the result before anything ships.

The anonymization record is delivered with the data, so a kitchen pretraining program has a clear paper trail from contributor consent through anonymization to the license terms governing commercial training on the finished corpus.

You describe the kitchen types, tasks, and objects your pretraining corpus is missing. That becomes a written spec covering the task list, the scene types it needs to span, the participant and lighting mix, capture tier, annotation depth, and accept criteria, quoted for reach, timeline, and price before any capture begins. Many programs start with a Spec Pilot, a small, fast first batch that proves the spec and accept criteria before scaling volume.

From there the collection runs through the same seven-stage end-to-end pipeline as every Firsthand program: contributor recruiting, capture across the specified kitchens, per-batch QA, anonymization, and licensed delivery, with rejected items recaptured at our cost and on-demand top-ups running against the same accept criteria afterward. A median 48-hour turnaround on a first batch means a kitchen pretraining program can validate its spec — scene coverage, object diversity, mask quality — quickly, before scaling the same pipeline to the volume the model needs.

Every custom collection is billed per validated hour and delivered under a perpetual, buyer-owned commercial license, so a kitchen pretraining corpus can be trained on and productized without the research-only or unstated licensing terms that most public egocentric video corpora carry.

Delivery happens in RLDS, LeRobot, and WebDataset, plus HDF5, zarr, and Rerun .rrd, with readable converter source so you can retarget the schema to your own loader rather than being locked into one format.