Egocentric Table Setting and Clearing Video Data

Table service is a short, repeated routine with real variation: how a stack is gripped, where each item lands, and in what order a table is cleared and wiped. The task catalog states this work directly. Its Hospitality Service Set row, Set table service, reads: "Lay a full place setting." A neighboring row, Clear + bus table, reads: "Stack dishes, bus, and wipe."

A first-person camera records that routine from the viewpoint a robot or an assistive system would act from, so the video shows what could be seen when each item was picked up and placed. Hand pose and contact events record the handling, and action segments mark where each step starts and ends. A manipulation policy can learn from this evidence without any robot being involved in the capture.

It helps to be exact about what the data is. It records a person setting and clearing a table: the reach, the placement and the order of steps. Place-setting layout labels, object identity labels and breakage labels are not stated on the site, so this page does not promise them.

Tabletop manipulation data treats a tabletop as a capture setting, not a skill: one work surface, a bounded workspace, and hands reaching across it. Object rearrangement data covers many objects moved from where they are to where they belong until a scene matches a goal arrangement. Dishwashing data covers washing, racking, drying and putting away dishes and clearing a kitchen after use.

This page is organised by the service task itself. It starts from two catalog rows that name a place setting and a bussed table, in a hospitality service environment, and treats laying and clearing a table as the unit of work rather than as one row in a broader list.

The dishwashing page lists Clear and bus a table as one task among six in its own table. This page is the deeper treatment of that service routine, and it stops where dishwashing begins: once dishes reach a wash station, the dishwashing page applies.

The task catalog lists two rows that name the routine directly. In the Hospitality Service Set, Set table service lays a full place setting and is rated Medium, and Clear + bus table stacks dishes, buses and wipes and is rated Easy. Both carry the status Available on request, which means the work is scoped and recruited to spec when a buyer asks for it.

The same environment lists neighboring service rows that a spec can combine with them. Fold napkins folds napkins to a set style, Pour + serve beverage pours a drink and serves it to a place, Tray assembly + delivery arranges, carries and sets a tray, and Wipe + reset counter clears, wipes and resets a counter. These are separate rows and are not described as table setting, but they sit in the same setting and often belong in the same episode list.

The first table below lists each row with its status. A catalog listing is a scoping aid, not a statement that every task has already been captured. Anything beyond the listed rows, such as a particular cutlery layout or a particular service style, is defined in the written spec.

The catalog places both rows in one environment, the Hospitality Service Set, which it describes as a table, a counter or bar, and a housekeeping or room area. That is an environment type, not a specific named venue.

The site classes hospitality as a thinner-coverage environment, where vetting operators for guest-facing and back-of-house roles is the bottleneck rather than demand for the task. It also states that front-of-house hospitality settings are bystander-dense, which lowers yield after de-identification. A table-service spec has to allow for both.

The site does not say that any table setting or clearing capture takes place inside a particular restaurant, hotel or client facility, and this page does not claim it. No restaurant, retailer or brand is named for any row.

The written spec is the contract, so it is worth being specific about the parts of table service the site leaves to the buyer. Name the table and the setting you want, for example a single place setting or a full table, and name the ware in scope: plates, glasses, cutlery and napkins, as your policy needs them.

Name the sequence that matters, such as laying the items in a set order, stacking dishes to carry them, or clearing and then wiping. Name the start state and the end state, and what counts as a finished episode: one place laid, a full table laid, or a table cleared and wiped.

Name whether spills, dropped items and re-placements should be captured alongside clean takes. Finally, name the capture mode: a staged service set recorded with handheld grippers for demonstrations that must map onto a robot end effector, or a different mode the quote confirms.

Both catalog rows name the same capture mode: a multiview rig with handheld grippers. A staged working set is recorded by a multiview rig while the operator performs the task with handheld grippers, so demonstrations map onto a robot end effector. The second table below shows this alongside the other capture mode the catalog names.

Each episode follows the same schema used across Firsthand custom collections: head and wrist RGB, depth, 3D hand pose, contact events and action segments, ten streams in all, aligned on one hardware clock and synced under 2 ms. Distance-invalid pixels beyond the reliable depth range are flagged rather than interpolated.

The site does not state a camera count, resolution, frame rate or lighting setup specific to table service, so those are agreed in the written spec rather than assumed. It also does not describe how depth behaves on glassware, polished plates or shiny cutlery, so raise that in the spec and test it in a Spec Pilot.

Hand pose tracks the operator through each reach, grip, placement and stack. Contact events mark when fingers make and break contact with an item or the table surface, and action segments mark the start and end of each step, such as pick up, place, stack and wipe. Because the labels share one clock with the video and depth, a step boundary lines up with the exact frame where it happens.

Instance masks are stated on the dishwashing, warehouse and object rearrangement pages, not on the hospitality page. If your policy needs masks for this task, say so in the spec and confirm them in the quote.

Object and ware labels use the terms your spec sets, so they follow the vocabulary your training stack expects rather than a generic taxonomy.

The site states that, once footage is captured, failure or recovery cases are captured and labeled where they occur, consistent with how action-labeled egocentric video is structured across the site. A slipped item or a re-placement belongs in that category, because a policy that only sees clean takes has no model of recovery.

The site does not give a rate for how often these cases occur, and this page does not give one either. Whether they should be kept in a batch is a spec decision.

Place-setting layout and goal-state labels are not stated, so the site does not describe a tagged result such as "correctly laid" or a per-item position on the table. Object identity or category labels, object 6-DoF pose, and breakage or outcome labels are also not stated.

Force and tactile streams are not stated, and neither is a camera or head 6-DoF pose. Item counts, cycle times and a throughput figure are not stated for this task family. If your policy needs any of these, they are defined in the written spec as annotation depth, and the quote covers them before capture begins.

Capture follows a documented consent chain for the operator wearing the rig, and the consent record is part of the delivery. The operator is the one wearing the capture hardware and giving consent.

Anyone else who appears in frame, such as a guest, a co-worker or a bystander, is handled through the site’s bystander and de-identification process rather than being captured as a second consenting participant. Faces and screens are blurred at ingest, a reviewer confirms the result before anything ships, and clips that cannot be reliably de-identified are discarded. The anonymization record is delivered with the data.

That is also why front-of-house settings yield less. Who may appear in frame is agreed in the spec before capture rather than assumed afterward.

Delivery formats are RLDS, LeRobot and WebDataset, plus HDF5, zarr and Rerun .rrd. The first three are the training-loader formats; HDF5 and zarr carry arrays for custom pipelines, and Rerun .rrd gives a multi-stream synchronized playback file for review, which suits visually checking a table episode before writing a loader.

Converter source is readable, so you can retarget the schema to your own loader, and the episode schema page lists the fields.

You describe the tables, ware and sequence your policy needs. We turn that into a written spec covering tasks, environments, capture tier, annotation depth and accept criteria, then quote reach, timeline and price against that document before any capture begins.

The collection runs through the same seven-stage end-to-end custom collection pipeline as every Firsthand program: contributor recruiting, capture, per-batch QA, anonymization and licensed delivery. Rejected items are recaptured at our cost.

Where coverage is already deep, the first validated batch lands in a median of 48 hours from signed spec. Hospitality is a thinner-coverage environment, where the site states a first batch typically takes three to four weeks because vetting operators is the bottleneck. We quote the timeline up front, and the quote tells you which case you are in.

On-demand top-ups run against the same spec and accept criteria after each training run, so new table data stays consistent with what you already hold. A Spec Pilot, a small and fast first batch, can prove the spec before you scale volume.

Every custom collection ships under a perpetual, buyer-owned commercial license, and exclusivity is available so the data is never resold. The license is documented alongside the consent chain, so a policy trained on the data can go into a product.