Egocentric Video Data for Food Service and Restaurant Robotics

Egocentric food service data is first-person video of real commercial food-handling work — recorded from a camera positioned near the worker’s own point of view, at a chest mount, head rig, or wrist camera, rather than a fixed overhead or side-angle camera watching the station. That viewpoint matters for training a robot or AI system to perform the same task, because it sees the work the way a robot’s own onboard camera would see it over a cutting board, a plating line, or a dish pit, rather than from an angle no robot will ever occupy.

The underlying capture, labeling, QA, and licensing process is the same one used across every custom collection on this site: a written skill spec describing the tasks and setting, paid and consented contributors performing the real work, synchronized sensor streams recorded alongside the video, and per-batch QA before delivery. What changes for food service is the setting and the task list written into that spec, not the process itself — the same relationship this site’s industry page describes between kitchen and food-prep capture and every other industry setting it covers.

Commercial food service covers a narrower, faster-paced set of tasks than a home kitchen: prep and cutting at volume, assembly and plating to a standard, cooking-line handling under time pressure, dishwashing and cleanup, and packaging for service or delivery. Each of those task families breaks down into specific, repeatable steps — the kind of structure a written skill spec is built to describe.

A good fit for capture is any task a restaurant or commercial kitchen performs the same way, by design, many times a shift: a standard plate build, a specific cut pattern, a fixed dish-pit sequence, a packaging routine for a delivery order. Highly improvised or one-off tasks are harder to specify cleanly and are usually a worse starting point than a task the kitchen already runs to a consistent method.

This differs from the pretraining angle the site’s general kitchen page covers, which is about broad manipulation diversity across many domestic and shared-house kitchens for a foundation model’s base layer. A commercial food service program is usually narrower and more operational: a specific restaurant, line, or task set a buyer needs modeled accurately, not the widest possible variety of kitchens.

A food service collection starts the same way any custom collection on this site does: you describe the restaurant or kitchen tasks your training pipeline needs covered — which stations, which dishes or task types, which steps matter, and which failure cases need to be represented. That becomes a written skill spec naming the task list, the setting, the participant mix, capture tier, annotation depth, and accept criteria, quoted for reach, timeline, and price before any capture begins.

Because a kitchen or service line is already a real operating environment, the spec also has to account for what is already true about that setting — shift patterns, station layout, which areas have customers in frame, and which equipment is already in use — rather than treating the space as a blank set to be arranged for the camera.

Many programs start with a Spec Pilot, a small, fast first batch that proves the spec and accept criteria on a limited set of tasks before scaling to full volume. That lets a buyer confirm coverage and label quality on, say, one plating station or one prep task before committing to capture across an entire line or multiple locations.

The same capture tiers used across every Firsthand collection apply to food service work: a phone in a chest mount for a fast, low-friction capture, up to a head rig or full multi-camera setup for richer stream coverage, chosen to match the spec’s requirements and the pace of the task being captured. Streams recorded alongside the video can include depth, IMU, and 3D hand pose, synchronized to the video under the same hardware-alignment standard used across the catalog.

A working kitchen or restaurant line presents constraints a quieter environment does not: wet, reflective, and hot surfaces that affect depth sensing, moving staff and equipment sharing the same tight space, and a pace that leaves little room for retakes mid-shift. Capture is planned around service hours and station layout rather than requiring the kitchen to pause normal operations for the camera.

Raw video alone does not tell a model what happened in a task, which is why captured footage is broken into the same structure used across every collection on this site: action segments marking the start and end of each discrete step, episodes grouping a full task attempt from start to finish, and hand pose tracking the fine manipulation that a video frame alone does not resolve.

A plating task, for example, segments into its component steps — portioning, placement, garnish, final check — rather than being labeled as one undifferentiated clip. That step-level structure is what lets a training pipeline learn the individual actions inside a task, not just recognize that “a task happened.”

Failure and recovery cases are part of that same labeled structure: a dropped item, a mis-cut, a redo of a plating step. Representing how a real worker catches and corrects a mistake, not only the clean version of a task, is part of what a written spec can call out as a requirement for the batch.

Every custom collection ships under a documented consent chain, with each episode linked to a signed contributor release. Faces, plates, and screens anywhere in frame are detected and irreversibly blurred at ingest, with a reviewer confirming the result before anything is delivered.

In a restaurant setting, that same anonymization standard applies to anyone who appears in frame, including customers and other staff who are not contributors to the collection — the same bystander-protection approach the site’s industry page describes for retail floors and other settings where people other than the task performer are present. A clip where someone in frame cannot be properly de-identified is handled the same way it is everywhere else on this site: discarded rather than shipped with a caveat.

The buyer. Every custom collection, including a food service program, is delivered under a perpetual, non-exclusive commercial license by default, with exclusivity available, so the finished dataset can be trained on and productized without a research-only or unstated licensing arrangement.

Delivery is billed and licensed the same way as any other collection on this site — against the accept criteria named in the written spec, with rejected items recaptured at no cost to the buyer before anything ships. Pricing is quoted per spec rather than published as a flat rate, because the right capture tier, annotation depth, and volume vary by program; see the pricing page for how a quote is built.

Food service data ships in the same formats available for any custom collection on this site: RLDS, LeRobot, and WebDataset, plus HDF5, zarr, and Rerun .rrd, or raw streams with metadata if your pipeline needs the unprocessed form. Readable converter source ships alongside the data, so the schema can be retargeted to your own loader rather than locking you into one format.

Describe the restaurant or kitchen tasks, stations, and failure cases your training pipeline is missing. That becomes a written spec covering the task list, setting, participant mix, capture tier, annotation depth, and accept criteria, quoted for reach, timeline, and price before capture begins.

From there the program runs through the same end-to-end pipeline as every Firsthand collection: contributor recruiting, capture in the working kitchen or restaurant setting, per-batch QA, anonymization, and licensed delivery, with on-demand top-ups available afterward against the same accept criteria. A median 48-hour turnaround on a first batch means a food service program can validate its spec on a limited task set before scaling to the volume a model needs.