Egocentric Cooking Video Data Collection

Egocentric cooking video data is first-person video of a person cooking or preparing food, recorded from the cook’s own point of view with synchronized sensor streams alongside. The camera sits on the head, close to what a robot’s onboard camera would see over a counter, so the frames show what the hands are doing to the ingredient, the tool and the pan rather than a distant view of someone standing in a kitchen. It is the same idea behind any first-person cooking dataset, applied here as a custom collection scoped to your tasks.

On this site the data is human capture. A paid, consented contributor wears the rig and performs a cooking or food-preparation task from a written skill spec. It is not robot or teleoperation data. This page covers buying a custom collection of those demonstrations. It is narrower than kitchen pretraining at scale, which has its own page, and than commercial food-service capture, which also has its own page. The subject here is cooking and food preparation itself: how it is specified, what makes it hard to record, and how it is labeled so a training pipeline can use it.

NVIDIA’s GR00T N1 paper describes a data pyramid with web data and human video at the base, synthetic data in the middle and real robot trajectories at the top. First-person human video of cooking sits in that base layer. It is far more abundant than robot data, and a person can move through many kitchens without the safety and access problems of taking an early-stage robot near knives, hot surfaces and hot liquids. That is the practical case for kitchen task video data for robots: the demonstrations come from people, in the kitchens where the robot will eventually work.

Public cooking corpora are captured for someone else’s task distribution. EPIC-KITCHENS-100 is the reference example: its paper describes 100 hours of unscripted footage from 45 kitchens, recorded with head-mounted cameras. Its authors also report that adding more data saturated and that the results highlight the need for more diverse data rather than merely more data. A custom cooking video collection lets you aim directly at the diversity your model is missing: the dishes, tools, kitchen types and participants a public release does not cover.

Generalist’s GEN-0 write-up makes the same point in different words: "data quality and diversity matters more than sheer volume." A cooking corpus that repeats the same counter and the same three recipes does not generalize better than one that visits many distinct kitchens once each. Collection is on demand, so a focused brief is a normal starting point, and you do not need a standing program to begin.

Cooking stacks several capture problems that a tabletop pick-and-place task does not have. Each one is already addressed by a stream or label this site states, and the table below maps each cooking challenge to the stream or label that addresses it.

First, occlusion by the hands themselves. A hand wrapped around a knife or a pan handle hides its own fingers, and the held object often hides the food. 3D hand pose is delivered for both hands with a per-joint visibility flag, contact events mark when the hand meets the item, and a batch is rejected if the manipulated object is out of frame for more than 15 percent of an episode.

Second, steam, heat and wet surfaces. A rinsed pan, a wet counter or a steel sink degrades depth, and steam can disrupt it for 2 to 6 seconds at a time. Per-frame depth confidence maps ship with the depth stream, steam is flagged at the episode level, and an episode with more than 35 percent invalid depth pixels in the manipulation region goes to the reject log. Heat itself is not sensed: no thermal stream is listed, and the stovetop transfer task is described as needing heat-adjacent care rather than being measured.

Third, deformable and liquid ingredients. Dough and produce change shape as they are handled, so they ship per-frame instance masks rather than propagated ones. Pouring and decanting exercise controlled release and liquid-object contact. The site does not list a liquid-level or fill-state label, so that is not promised.

Fourth, long multi-step recipes. A recipe is a chain of dependent steps, and an early mistake changes everything after it. That is why a cooking spec writes the steps and their order down, why each step becomes an action segment, and why failure and recovery cases can be required in the batch.

A cooking collection uses the same capture stack as every Firsthand program. The instrumented tier is a head-mounted stereo rig that records first-person RGB, with a metric depth stream recovered from the calibrated stereo rig rather than from a standalone depth sensor, and 6-DoF IMU data at 200 Hz from head- and wrist-mounted sensors. The wider stack is ten streams: a 4K head camera, two wrist cameras, depth, IMU, 3D hand and body pose, instance segmentation and action labels, hardware-aligned under 2 ms.

Every stream is hardware-triggered onto one PTP clock domain, so depth, motion and hand pose line up with the same video frame rather than being approximated afterwards. The rig is calibrated twice per session, before capture starts and after it ends, and an episode is accepted only if RMS reprojection stays under 0.28 px. The two wrist cameras give a second, closer view of the hands than the head camera alone.

Each attempt is delivered as one episode. Inside it ship action segments, 3D hand pose with visibility flags, contact events, objects and per-frame instance masks, plus depth confidence maps for kitchens. Force and tactile readings are not among the streams listed here, and no robot or teleoperation data is captured.

You describe the cooking or food-preparation task your pipeline needs covered, and that becomes a written skill spec. For a recipe, the spec names the task and its sub-steps, their order, the kitchen type, and the ingredients and tools involved. It defines where an attempt starts and ends and what counts as a completed attempt, such as a finished dice or a transferred pan. It also sets the capture tier, annotation depth and accept criteria, and states how much failure and recovery coverage to include alongside clean runs. Our skill spec guide walks through how to write one.

The kitchen task categories this site already documents are knife work (dice, julienne, chiffonade), pouring and decanting between vessels, opening jars, bottles and vacuum-sealed packaging, stovetop transfer with a loaded pan, wiping and resetting a work surface, loading and unloading a dishwasher, and bimanual dough handling. They are captured in domestic kitchens, shared-house kitchens and small commercial prep lines. A dish or technique outside that list is a skill-spec question to settle in a Spec Pilot rather than an assumption.

Three spec choices matter most for cooking. Kitchen types should be named, because each presents a different counter layout and clutter pattern. Participants should be varied, since left- and right-handed knife work, different chopping styles and different pours keep a model from overfitting to one person’s technique. And lighting should be named: daylight coverage fills fastest, while evening, low-light capture is the harder cell to fill. How many kitchens, contributors and hours a program covers is set per program in the skill spec.

Action segments divide an episode into start- and end-timestamped spans, one per sub-step, each carrying start_ns, end_ns, verb and noun. For cooking, a segment might be one cut, one pour or one transfer, as written in the skill spec. The other standard fields are hand_joints and hand_visible for 3D hand pose, contact_events for the moment the hand meets the item, objects, and instance_masks, stored at the depth resolution of 848×480 so they line up with the depth stream.

Validation runs against signed accept criteria written into the spec before capture. Each batch goes through per-batch QA, and a reject log ships with it. Episodes are machine-checked against stated reject criteria: a sync breach beyond the ceiling for more than three consecutive frames, calibration drift beyond tolerance, the manipulated object out of frame for more than 15 percent of the episode, more than 35 percent invalid depth pixels in the manipulation region, an undocumented bystander, or a mismatch against the named skill spec. A human reviewer then checks each episode against the task spec.

Only footage that clears the criteria is billed as a validated hour, and rejected items are recaptured at our cost. Many teams start with a Spec Pilot, a small, fast first batch that proves the spec and the accept criteria on real cooking footage before volume scales.

Kitchen capture happens inside lived-in spaces, which raises the bar above a workshop capture. Every contributor signs a written release before capture, and each delivered episode links to that record through consent artefact IDs in the batch data card. The consent chain has six checkpoints, from before capture to after delivery, and a break at any one discards the footage rather than shipping it at a lower grade.

At ingest, automated detection finds faces, plates and screens and blurs them irreversibly, and speech is scrubbed of names, addresses and card numbers while ambient audio is kept. A human reviewer confirms that no un-consented person is identifiable in any frame. If a person cannot be de-identified, the footage is discarded. Site permission is kept on file where the site needs it, and the exact permission detail for your kitchens is set per program in the skill spec. The anonymization record is delivered with the data. None of this is legal advice, and the executed agreement controls.

Where this site does not state something, this page says "not stated" rather than filling the gap. The table below separates what is delivered from what is not stated. There is no listed label for food state such as doneness or temperature, no liquid-level or fill-state label, no recipe or ingredient ontology beyond the verb and noun on each action segment, and no force, tactile or thermal stream. Nothing here claims a training outcome.

The tradeoff to plan for is the embodiment gap between a human hand and a robot end-effector, which retargeting or pairing with robot data has to close. What you get is human first-person demonstrations in which the hands, the food and the step structure are all explicit. Counts of kitchens, contributors, recipes and hours are not fixed numbers on this site; they are set in your skill spec.

Public datasets are a useful baseline, and the comparison below uses EPIC-KITCHENS-100, taking each fact from its paper and official site. The structural difference is control. A public release has fixed scope set by its authors, while a custom collection is scoped to the dishes, tools, kitchens and participants your model is missing, with the consent, anonymization and license terms written into your agreement. Whatever dataset you use, read its license and consent terms before training on it.

Delivery is in the format your stack already reads: RLDS, LeRobot, WebDataset, HDF5, zarr or Rerun .rrd, or raw streams with metadata. Readable converter source ships alongside the data so the schema can be retargeted to your own loader. Collections are billed per validated hour.

The license is perpetual, irrevocable and worldwide, with the data buyer-owned and non-exclusive by default. Training, evaluating, fine-tuning and deploying models, including commercial ones, is permitted, and exclusivity is available as a separate term. The executed agreement controls.

From signed spec to first validated batch, the timeline is a median of 48 hours where coverage is deep and three to four weeks where it is thin and contributors need to be recruited first. After the first batch, on-demand top-ups run against the same accept criteria, so you can add more dishes, kitchens or variation without rewriting the spec.

Describe the dishes or food-preparation tasks, the kitchen types, the sub-steps and their order, and the failure or variation cases your pipeline is missing. That becomes a written skill spec, quoted for reach, timeline and price before any capture begins, and most programs run a Spec Pilot first. A published home-cooking sample recording is on the sample data outputs page if you want to see the streams before you write a spec.