Egocentric Beverage and Coffee Prep Video Data

Egocentric beverage and coffee prep video data is real-world, first-person video of a person making drinks: grinding beans, dosing and tamping, pulling espresso shots, steaming milk, pouring, building and garnishing a bar drink, and operating a coffee machine. It is recorded from the operator viewpoint, with depth, 3D hand pose, contact events and frame-accurate action segments aligned on one hardware clock.

Drink making asks a lot of a manipulation policy because it is a sequence, not a single motion. The hands switch between tools, a vessel has to be positioned under a spout or a kettle before anything moves, and the result is handed to someone else. A first-person view keeps the vessel, both hands and the equipment in the same frame, which is the viewpoint a robot arm with a head or wrist camera would act from.

This page treats beverage prep as a task family in its own right. Pouring and liquid handling is about the motion of moving liquid, and cooking is about preparing food. Here the finished object is a drink: assembled, made on a machine or at a bar, and served.

The task catalog already lists beverage work in several settings. In a cafe, bar and beverage service environment it lists one row, Professional Barista Workflows: grind beans, dose and tamp, pull espresso shots, steam milk and pour drinks at the bar. In a staged working kitchen set it lists pour-over coffee: grind, wet the filter and pour a coffee. A staged hospitality service set lists two rows, pour and serve a beverage, and prepare and garnish a bar drink. An office and professional services environment lists coffee machine operation: fill the machine with water and beans or pods, select the drink and serve the cup.

The catalog also lists nearby rows that touch drinks without being beverage prep as their main task. A care room row positions a drink with a straw. A hotel row restocks toiletries, towels, coffee and minibar items from the housekeeping cart, and a nursing home row sets up a meal tray and helps a resident eat with a spoon and cup.

A custom spec is scoped from that same vocabulary. You describe the drinks, equipment, stations and sequence your policy needs, and the written spec covers them, with the catalog tasks as a starting point rather than a fixed menu. The first two tables list every row with its description, environment, skill rating and status as the catalog states them.

The most complete beverage row is Professional Barista Workflows. It is described as grinding beans, dosing and tamping, pulling espresso shots, steaming milk and pouring drinks at the bar. It is rated Medium, it is recorded with a head-mounted phone and a multiview rig, and its status is Available on request.

It sits in the Cafe, Bar & Beverage Service environment, whose examples are a coffee shop, a juice bar, a cocktail bar and a hotel bar. The catalog lists that environment with this one task row. The site does not state which specific venues a collection runs in, so the environment, access and schedule are agreed in the written spec.

Because the row chains several steps, action segments matter here. A spec can ask for the whole workflow or isolate one step, and the segment boundaries mark where grinding stops and dosing starts, where tamping gives way to the shot and where steaming gives way to the pour.

Pour-over coffee is described as grinding, wetting the filter and pouring a coffee. It is rated Medium and is Available on request. It is a staged row, recorded by a multiview rig while the operator uses handheld grippers, in a Working Kitchen Set described as a stovetop, counter, sink, dishwasher, utensils and food props.

The grippers matter for training. The catalog describes this capture mode as one where demonstrations map onto a robot end effector, so the recorded grip and the motion of the pour are made with a tool the robot can reproduce rather than a bare hand.

Pouring is also covered as a motion on its own page, linked below. This row is the one that places that motion inside a beverage task, with the grind and the filter step before it.

Two rows sit in a Hospitality Service Set, described as a table, a counter or bar and a housekeeping or room area. Pour and serve beverage is described as pouring a drink and serving it to a place. Prepare and garnish drink is described as building and garnishing a bar drink. Both are rated Medium, both are recorded by a multiview rig with handheld grippers, and both are Available on request.

The first row ends with a hand-off to a place at a table or the bar, so the serve step is part of the demonstration. The second row ends with a garnish, which is a small fine-motor step at the end of a longer build. A spec can ask for either on its own or the full sequence from pour to serve.

The site does not state a drink list, recipes or cup and glass sizes, so which drinks are made and in what vessels is agreed in the spec rather than assumed.

Coffee machine operation is described as filling the machine with water and beans or pods, selecting the drink and serving the cup. It is rated Easy, it is recorded with a head-mounted phone and a multiview rig, and its status is Available on request.

It sits in the Office & Professional Services environment, whose examples are a corporate office, a co-working space and a reception area, with a coffee machine named among the desk work, printing, filing and scanning examples. This is a different kind of beverage task from the barista row: the operator loads and selects, and the machine does the making.

The site does not name a machine brand or model and does not state button, display or menu data for any machine. If your policy needs the machine interface in frame, say so in the spec so the camera placement is agreed before capture.

One row serves a drink to a person. Serve drink with a straw, in a Care Room set, is described as positioning a drink with a straw. It is rated Easy, recorded by a multiview rig with handheld grippers, and Available on request. The care room examples list a bed, a tray table, linens and a participant or mannequin.

It is a short placement task, but it is the hand-off step that ends many beverage sequences, and it is the closest catalog row to a drink being brought to someone. This page does not present it as a care or assistive-feeding dataset, and the site makes no clinical claim for it.

The catalog places these rows in a cafe, bar and beverage service environment, a staged working kitchen set, a staged hospitality service set, an office and professional services environment and a staged care room. Each task row carries a status that says how soon capture can begin.

The site defines those statuses plainly. Ready to deploy means contributors and site access are set up and capture starts once a spec is signed. Active collection means recording is running now, so a spec can join an existing capture stream. Available on request means the work is scoped and recruited to spec when a buyer asks for it. Every beverage row in the current catalog is Available on request, so the timeline is the scoped-to-spec case rather than the ready-now case.

The site does not state a named cafe, coffee chain, appliance or machine brand for any row, and it does not state capture at a customer own facility. Environment, access and schedule are agreed in the written spec.

The catalog names two capture modes. In the first, a head-mounted phone records the first-person view of real work on site, and a multiview rig adds synchronized third-person angles when the spec asks for them. In the second, a staged working set is recorded by a multiview rig while the operator performs the task with handheld grippers, so demonstrations map onto a robot end effector.

A head-mounted camera keeps the grind, pour and serve motion in frame from the viewpoint a manipulation policy will see from, with a wrist-mounted camera available alongside it for close-range detail. Depth, 3D hand pose, contact events and action segments are aligned to video on one hardware clock, synced under 2 ms. Distance-invalid pixels beyond the depth sensor reliable range are flagged rather than interpolated.

The site does not state a camera count, resolution or frame rate specific to beverage prep, so those are agreed in the written spec rather than assumed.

Each episode follows the same episode schema used across Firsthand custom collections: ten streams synced under 2 ms, with head and wrist RGB, depth, 3D hand pose with per-joint visibility, contact and grasp events, and frame-accurate action segments.

For beverage prep, hand pose tracks the operator through the grip, the tilt and the placement, contact events mark when the fingers make and break contact with a cup, a tool or a control, and action segments mark the start and end of each step in the sequence. Because the labels share one clock with the video and depth, a step boundary lines up with the exact frame where it happens.

Object labels use the terms your spec sets. The skill spec decides which object classes are in scope before capture, so labels follow the vocabulary your training stack expects rather than a generic taxonomy.

Cup or vessel sizes, liquid volumes or fill-level labels, drink recipes, throughput, force or tactile data, temperature data, a task-completion label and any food-safety or hygiene certification are not stated anywhere on the site as standard fields. Some catalog task names mention espresso, steam or garnish, but those describe what the operator does, not a field delivered with the episode.

If your policy needs any of these, they are defined in the written spec as annotation depth, and the quote covers them before capture begins. The page does not promise them by default, and it makes no food-safety or hygiene claim.

Delivery formats are RLDS, LeRobot and WebDataset, plus HDF5, zarr and Rerun .rrd. The first three are the training-loader formats; HDF5 and zarr carry arrays for custom pipelines, and Rerun .rrd gives a multi-stream synchronized playback file for review. The format table below sets each against what its official documentation states.

Converter source is readable, so you can retarget the schema to your own loader, and the episode schema page lists the fields. Beverage prep episodes use the same schema as every other Firsthand collection, so a loader that already reads another Firsthand dataset reads this one without changes.

Every batch is validated against signed accept criteria before it ships, with a reject log, and delivery is rolling, per validated batch.

Every episode links to a signed contributor release through a documented consent chain, and the consent record is part of the delivery. Contributors are the people who make the drinks, so their release covers the footage captured of their hands and workspace.

Faces, plates and screens are detected and irreversibly blurred at ingest, and a reviewer confirms the result before anything ships. The anonymization record is delivered with the data. A cafe, a bar or an office can put other people in frame, which is why that pass matters here.

Text on cups, labels and packaging is not listed as a blur category on the site, so this page does not claim it is blurred. If text on a vessel or a package matters for your use, raise it in the spec so the handling is agreed before capture.

Where coverage is already deep, the first validated batch lands in a median of 48 hours from signed spec. Thin environments typically take three to four weeks, because vetting operators there is the bottleneck, and we quote that timeline up front. The catalog status on each task row tells you which case you are in, and every beverage row is currently Available on request.

On-demand top-ups run against the same spec and accept criteria after each training run, so new beverage prep data stays consistent with what you already hold. The site does not state a batch size or a later-batch turnaround, so those are quoted against your spec.

You describe the drinks, equipment, stations and sequence your policy needs. We turn that into a written spec covering tasks, environments, capture tier, annotation depth and accept criteria, then quote reach, timeline and price against that document before any capture begins.

A Spec Pilot, a small and fast first batch, can prove the spec and the accept criteria before you scale volume. The collection runs through the same seven-stage end-to-end custom collection pipeline as every Firsthand program: contributor recruiting, capture, per-batch QA, anonymization and licensed delivery. Billing is per validated hour, a reject log ships with every batch, and rejected items are recaptured at our cost.

Every custom collection ships under a perpetual, buyer-owned commercial license, and exclusivity is available so the data is never resold. The license is documented alongside the consent chain, so a policy trained on the data can go into a product.