Egocentric Object Rearrangement and Tidying Data

Object rearrangement is the task of changing a scene from one arrangement of objects to another. A person or a robot moves several objects, one at a time or in groups, from their current positions to target positions, and the task is finished when the scene matches the target arrangement. Tidying is the everyday form of it: the target is a conventional arrangement, such as items in their storage place, objects on a shelf in order or a surface cleared, rather than a set of coordinates.

Rearrangement and tidying data is a recording of that work together with the signals a policy learns from: where the hands go, which object is being handled, when contact is made and broken, and how the task divides into steps. This page treats rearrangement as a task family and says plainly which labels the Firsthand site states and which it does not.

The site already has neighbouring pages, and each is organized by a different axis. Egocentric pick and place data is one manipulation primitive: reach, grasp, transport and place of an object. Egocentric tabletop manipulation data is a capture setting: one bounded work surface. In-home data collection and household robot pretraining data are organized by setting and training stage, and both list tidying as one chore among laundry, cleaning and surface reset.

Rearrangement is the task family that cuts across all of them. It is built from many pick-and-place actions chained over many objects, it can run on a tabletop, a shelf, a rack or a whole room, and it is defined by its end state as much as by its motions. The questions it raises are its own: what the goal arrangement is, how many objects an episode moves, in what order, and how an object stays identifiable from the first touch to the last placement. The first table sets the pages side by side.

The site does not publish a rearrangement-specific task list, so this page points to the tasks it already describes and names where each comes from. In the household setting, the chores dataset lists surface wipe and clutter reset among its six stated tasks, and the in-home page describes household tasks as long-horizon, low-precision and navigation-heavy.

Outside the home, the task catalog behind the environments hub lists tasks in the same family: re-hanging and tidying a fitting room, tidying and resetting a care room, preparing a room, and returning weights to a rack in order. The retail page lists rearranging apparel on a fixture. The second table sets these out. Each catalog row carries a sector, environment, task, description, skill level, capture setup and status. A listing is a scoping aid, not a statement that the task has been captured.

A rearrangement task has a start state, a sequence of moves and a goal state. In tidying the goal is usually implicit: a person knows where things belong, so a spec has to say it in words. The site states that every skill spec names the tasks, the setting, the object set, the capture tier, the annotation depth and the accept criteria, and that a pick-and-place spec is scoped by describing the objects, the start and goal locations and the setting.

The site states no goal-state label, no before-and-after scene capture and no scene-reset protocol. What it does state is how an episode is bounded: action segments carry start and end to the frame, and every batch is validated against signed accept criteria. Writing the start state and the goal arrangement into the spec is therefore a sensible item for the brief, and anything stronger, such as an image of the target arrangement or a list of target positions, is a conversation to have per program. The table headed “What does the site state about rearrangement data?” marks each of these as stated or not stated.

Rearranging a room is many small actions. Each object is its own reach, grasp, transport and place, and the episode adds decisions on top: which object next, where it goes, and whether to move one thing to reach another. That makes episodes longer than a single-primitive clip, and it is why the site frames household tasks as long-horizon and navigation-heavy rather than as single grasps.

That has three consequences for the data. The episode has to be segmented, so a model can see each sub-action and its order. The data has to span many objects and scenes, because a guide on this site notes that tidying a counter full of different items needs more object and scene diversity than a task on one consistent object. And the whole arc has to be captured, because the in-home spec names the full arc and the rooms it passes through.

Every Firsthand custom collection uses the same streams and labels, and the third table lists them for this task family. Hand pose is 21 keypoints per hand in 3D with a per-joint visibility flag, and body pose is 24 joints. Contact events mark grasp and release. Action segments use a verb and noun taxonomy of 97 verbs and 512 nouns, with start and end to the frame, human-reviewed.

In a rearrangement episode, contact events give the boundary of each pick and each place, action segments give the structure of the whole sequence, and hand pose carries how each object was held. Which verbs and nouns a rearrangement spec uses is set per program; the site does not list a rearrangement vocabulary.

Rearrangement is the task where the object matters most, because the same hand motion moves a mug or a book and the policy has to know which. The site states instance masks on manipulated objects and target surfaces, with tracked IDs across the episode, linked to an objects list and to contact events. A guide on this site adds that a spec attaches instance masks where object identity matters to the policy, and leaves them out where it does not.

Object pose, a scene graph and an explicit category label per object are not stated on the site, and this page does not claim them. If your policy needs object pose or a structured description of the scene, say so in the brief; whether and how it is attached is confirmed per program.

The rig is worn by the contributor. The site states a head-mounted global-shutter camera at 3840 × 2160 and 60 fps with a 120° horizontal field of view, two wrist-mounted cameras at 1920 × 1080 and 60 fps with a 100° horizontal field of view, a depth stream with a per-frame confidence map, and a 6-DoF IMU at 200 Hz. The ten streams are hardware-triggered onto one clock and aligned under 2 ms.

Intrinsics and extrinsics are solved twice per session, before capture starts and after it ends, with a reprojection gate of 0.28 px, and the drift between the two solves is written into the episode metadata. A fixed rig is not stated, and no robot or teleoperation capture is part of this page.

Validation runs against signed accept criteria written into the spec before capture, and a reject log ships with every batch. The fourth table lists the machine-checked criteria the site states and what each means here. A failing episode goes to the reject log and is recaptured at our cost rather than shipped with a caveat.

Failures are part of the data. Missed grasps, dropped items and mis-placed objects are captured and action-segmented alongside clean takes, and they count toward validated hours. In rearrangement a dropped or misplaced object usually has to be picked up again, so the recovery stays in the sequence rather than being cut from it.

You describe the rooms or surfaces, the objects, the start state and the goal arrangement your policy needs, and how many moves each episode should chain. We turn that into a written spec covering tasks, environments, capture tier, annotation depth and accept criteria, then quote reach, timeline and price against that document before any capture begins.

A Spec Pilot, a small, fast first batch, can prove the spec and the accept criteria on real footage before you scale volume. Anything this page does not state, such as home or contributor counts, is set per program in the skill spec.

Where coverage is already deep, the first validated batch lands in a median of 48 hours from signed spec. Thin environments typically take three to four weeks, because vetting operators there is the bottleneck, and we quote that timeline up front.

On-demand top-ups run against the same spec and accept criteria after each training run, so new rearrangement data stays consistent with what you already hold. Billing is per validated hour.

Every custom collection ships under a perpetual, buyer-owned commercial license, and exclusivity is available so the data is never resold. Each episode links to a signed contributor release through a documented consent chain, and bathroom capture excludes any episode with a person in frame other than the consenting operator.

Delivery formats are RLDS, LeRobot and WebDataset, plus HDF5, zarr and Rerun .rrd, with the anonymization record included. The executed agreement controls.