Egocentric Tabletop Manipulation Data

Tabletop manipulation is manipulation performed on a single, bounded work surface: the actor stays roughly in one place, the objects sit within arm’s reach, and the task is carried out by reaching, grasping, moving and placing things across that surface. Tabletop manipulation data is a recording of those tasks, together with the signals a policy learns from.

It describes a capture setting, not a skill. Pick and place, sorting, tool hand-off and two-handed assembly can all happen on a tabletop, and the same skills can also happen on a shelf, in a cart or across a room. This page covers the setting: what is in frame, what has to be written into the spec, and what the Firsthand site does and does not state about it.

Those pages are organized by manipulation primitive or by hand articulation. Egocentric pick and place data is the reach, grasp, transport and place motion. Egocentric grasping data is the grasp itself. Dexterous manipulation data is finger-level work, and bimanual manipulation data is two hands acting together. Each names something the hands do.

Tabletop is the other axis: where the work happens. A buyer training a tabletop policy usually needs several of those primitives at once, inside one bounded workspace, with the objects, the start and goal positions and the surface itself under control. The questions that matter are different: how the work surface is framed, how the object set is chosen, how each take starts, and how the setup lines up with the robot’s own workspace.

The Firsthand site does not publish a tabletop-specific task list, so this page names families by the existing pages they map to rather than inventing a catalog. Pick and place, grasping, tool use and assembly, two-handed work, fine finger manipulation and failure and recovery can all run on a work surface, and each already has its own page with the labels it carries.

Where the site describes a family in a bench or kitchen setting, such as tool hand-off at a bench or food prep handling in a kitchen, the table says so. Treat it as a scoping aid: the tasks you need are named in the written skill spec, and anything the site does not state is set per program.

The rig is worn, so framing follows the contributor’s head. The site states a head-mounted global-shutter camera at 3840 × 2160 and 60 fps with a 120° horizontal field of view, which keeps the work surface and the reaching hands in view from the same viewpoint a tabletop policy would act from. Two wrist-mounted cameras at 1920 × 1080 and 60 fps, 100° horizontal field of view, add a close-range view of where the hand meets the object.

Depth comes from an active-stereo sensor at 848 × 480 and 30 fps with a stated range of 0.3 to 4 m. The site does not state a standard camera height, table size, working distance or framing protocol, so any of those that matter to your robot are written into the spec rather than assumed.

Matching the viewpoint to a robot is the reason framing matters. In the published EgoScale work, the aligned human demonstrations were collected with the same camera configuration as the robot, with matched viewpoints and calibrated intrinsics. The Firsthand site states that intrinsics and extrinsics are re-solved with a checkerboard target at the start and end of every session, so drift is detectable; whether a rig should match a particular robot is a spec conversation.

Every Firsthand custom collection starts with a written skill spec that names the tasks, the setting, the object set, the capture tier, the annotation depth and the accept criteria. For tabletop work the object set carries the most weight, because it decides what the policy sees across the surface.

The site publishes no standing tabletop object list, so the set is named per program, along with start and goal locations. On the labeling side, the site states instance masks on manipulated objects and target surfaces, with tracked IDs across the episode, so each object and the surface it lands on stay identifiable through the task. Who supplies the objects is not stated.

The site states no scene-reset protocol, so this page does not describe one. One term needs care: the household page lists surface reset as a chore category, meaning tidying a surface as the task itself. It is not a between-take reset procedure.

What the site does state is how a take is bounded and checked. Action segments carry start and end to the frame, every batch is validated against the signed accept criteria with a reject log, and rejected items are recaptured at Firsthand’s cost. A start-state definition is a reasonable item to put in the spec. For context, EgoScale’s robot evaluations used an image overlay of the target initial scene configuration to reduce variability in initial conditions; that describes evaluation in that paper, not a Firsthand practice.

The same streams and labels as every Firsthand custom collection: ten streams hardware-triggered onto one clock and aligned under 2 ms. Hand pose is 21 keypoints per hand in 3D with a per-joint visibility flag, and contact events mark grasp and release. Body pose is 24 joints. Action segments use a verb and noun taxonomy of 97 verbs and 512 nouns, start and end to the frame, human-reviewed.

The second table lists each stream with its stated spec and why it matters on a work surface. Depth carries a per-frame confidence map, and pixels beyond the depth sensor’s reliable range are flagged rather than interpolated, so a surface near the edge of the range is marked, not guessed.

Missed grasps, dropped items and mis-placed objects are captured and action-segmented alongside clean takes rather than discarded, and they count toward validated hours. On a work surface that matters because a policy trained only on clean takes has no model of recovery.

Failure and recovery episodes are labeled against the same accept criteria as clean takes. The site does not publish a tabletop-specific failure list; failure scenarios are named in the spec before capture starts, so each becomes a coverage cell that can be counted and checked.

You describe the tasks, the work surface and any workspace constraints, the objects, and the start and goal locations your policy needs. We turn that into a written spec covering tasks, environments, capture tier, annotation depth and accept criteria, then quote reach, timeline and price against that document before any capture begins.

The collection runs through the same seven-stage end-to-end pipeline as every Firsthand program: contributor recruiting, capture, per-batch QA, anonymization and licensed delivery. A Spec Pilot, a small, fast first batch, can prove the spec before you scale volume.

The EgoScale paper is a useful reference because its aligned mid-training dataset is tabletop by design. It comprises 344 tabletop manipulation tasks, each captured in approximately 30 human trajectories and 5 robot trajectories, totaling about 50 hours of human data and 4 hours of robot data. The authors describe it as focusing on tabletop tasks that match the robot’s workspace and kinematics.

That paper collected its data with a different stack from the one described on this site: motion-capture trackers and gloves for human hand motion, and teleoperated robot data. The Firsthand site states no robot capture, no teleoperation and no motion-capture glove capture, and this page implies none. The third table sets the two side by side. The paper is cited as published research; no relationship with its authors is implied.

Where coverage is already deep, the first validated batch lands in a median of 48 hours from signed spec. Thin environments typically take three to four weeks, because vetting operators there is the bottleneck, and we quote that timeline up front.

On-demand top-ups run against the same spec and accept criteria after each training run, so new tabletop data stays consistent with what you already hold.

Every custom collection ships under a perpetual, buyer-owned commercial license, and exclusivity is available so the data is never resold. Each episode links to a signed contributor release through a documented consent chain.

Faces, plates and screens are detected and irreversibly blurred at ingest, and a reviewer confirms the result before anything ships. Delivery formats are RLDS, LeRobot and WebDataset, plus HDF5, zarr and Rerun .rrd, with the anonymization record included.