Egocentric Mobile Manipulation Data Collection

Egocentric mobile manipulation data is first-person video of a person completing a task that takes them from one place to another and has them handle objects along the way, recorded from their own point of view with synchronized sensor streams alongside. The defining feature is that locomotion and manipulation share one episode instead of being filmed as separate clips.

This page covers buying a custom collection of those demonstrations. It is narrower than a general long-horizon collection, which is about long multi-step tasks of any kind, and than household pretraining at scale. The subject here is the combination: how a task that moves between locations is specified, labeled and delivered so a training pipeline can use it.

A tabletop task is short and local: reach, grasp, place, done, inside the view of a fixed workspace. A mobile task stretches that across space. The scene changes as the operator moves, the object being carried is in hand while the surroundings change, and the next place to act is not yet in view when the previous one finishes.

This site makes the same point about curated footage. Transit segments, such as carrying a tool between floors, are described as exactly the footage that staged datasets cut, and as a reason policies fail to move between rooms. A sequence of three sub-steps and a room transition is described as the kind of sequence a policy has to hold context across.

None of that is a claim about any trained model. The tradeoff to plan for is that these are human demonstrations, so using them for a specific robot means bridging the embodiment gap, for example by retargeting or pairing with robot data. What you get is first-person episodes in which the hands, the objects, the move and the step structure are all explicit.

Only tasks the site already states as combining movement between locations with hand manipulation are covered here. In the home: a laundry cycle, described as spanning a hamper, a hallway, a machine and a folding surface; vacuuming with obstacle negotiation, described as whole-room navigation with dynamic obstacle avoidance; and bin change and bag tie, described as fine manipulation combined with navigation. Household settings are listed as living space, bedroom, bathroom, utility room, and stairs and hallway.

In logistics, the site describes a worker who moves between stations during a shift, with action segments marking the transition points: leaving a pick aisle, arriving at a pack bench, feeding a sortation line. The site’s sample footage also shows an operator carrying a cordless vacuum down a staircase into a living room, and filling a container at a sink, crossing the room and watering plants.

The site does not list outdoor travel, vehicles, multi-floor facilities or other routes, and this page does not add them. If your task is not on the list, treat it as a skill-spec question to settle in a Spec Pilot rather than an assumption. Capture is not done at your own facility.

You describe the task your pipeline needs covered and that becomes a written skill spec. For a mobile task it names the locations the operator passes through, the sub-steps at each one and their order, and the items involved. It defines where an attempt starts and ends and what counts as a completed attempt, such as a finished laundry cycle or a bin changed and tied.

The move itself is a spec decision. A spec can name it as a sub-step with its own verb and noun, in the same way the logistics page marks the points where an operator leaves one station and arrives at the next, so the transition is a labeled span rather than an unlabeled gap. The spec also sets the capture tier, annotation depth and accept criteria, and states how much failure and recovery coverage to include.

It is quoted for reach, timeline and price before any capture begins, and the accept criteria are signed into it. Acceptance is measured against what was agreed rather than judged afterwards. The skill spec guide linked below walks through how to write one.

Paid, consented contributors perform the real task against the spec while wearing the capture rig. The instrumented tier is a head-mounted stereo rig: first-person RGB, with a metric depth stream recovered from the calibrated stereo rig rather than from a standalone depth sensor, and 6-DoF IMU data at 200 Hz from head- and wrist-mounted sensors.

Every stream is hardware-triggered onto one PTP clock domain, with int64 nanosecond timestamps per stream, so depth, motion and hand pose are aligned to the same video frame rather than approximated afterwards. The rig is calibrated before capture and again after the session ends. Wrist views are delivered as their own streams alongside the head view.

Collection is on demand, so you do not need a standing program, and a focused brief is a normal starting point. Force and tactile readings are not among the streams listed, and this page does not describe a navigation map, a localization output or leg and foot tracking, because none is listed among the delivered fields.

Each attempt is one episode. Inside it, action segments divide the task into start- and end-timestamped spans, each carrying start_ns, end_ns, verb and noun, so the order of stages and the transitions between them are in the data. The other fields are the standard episode schema: hand_joints and hand_visible for 3D hand pose across the whole sequence with a per-joint visibility flag, contact_events for the moment the hand meets an object, and objects.

Where the spec calls for it, per-frame instance masks are delivered at the depth resolution, 848×480. The head and wrist IMU streams carry the motion record that goes with the video. None of these are extra fields specific to mobile capture; the table below maps each label to its delivery field as listed on the episode schema page.

Teams building humanoids often search for loco-manipulation data, meaning a robot that walks and manipulates in one behavior. What this page delivers is the human side of that picture: first-person demonstrations of a person doing a task that spans locations. It is not robot locomotion data, not teleoperation data and not motion-capture data, and Firsthand does not capture robot or teleoperation data.

The site’s humanoid pages describe egocentric human video as one input to humanoid training. How a given policy uses these episodes, whether for pretraining, mid-training or alignment, is your pipeline’s decision. A spec can say which stages of the task matter most, so the episodes are weighted toward what your model is missing.

The site ships transitions and hard cases uncut, including walking between rooms, exposure hunting in dim light and occluded grasps, rather than curating footage down to the clean segments. Failure and recovery cases are part of the same labeled structure as a clean attempt, and a written spec can require them in the batch. Our failure and recovery page covers this in more depth.

Acceptance is measured against signed accept criteria written into the spec before capture begins. Each batch goes through per-batch QA, and a reject log ships with every batch. An episode is accepted only if RMS reprojection stays under 0.28 px and the drift is within tolerance; otherwise it goes to the reject log.

Only footage that clears the criteria is billed as a validated hour, and rejected items are recaptured at our cost. A reviewer can check that segment boundaries land on the actual contact moment rather than drifting into transit. Many teams start with a Spec Pilot: a small, fast first batch that proves the spec and the accept criteria before you scale volume.

The buyer owns it. Every custom collection is delivered under a perpetual, non-exclusive commercial license by default, with exclusivity available, so the dataset can be trained on and productized without a research-only or unstated arrangement. Every episode is linked to a signed contributor release under a documented consent chain, and faces and screens in frame are detected and irreversibly blurred at ingest, with a reviewer confirming the result before anything ships. Clips that cannot be de-identified are discarded rather than delivered.

Delivery is in the format your stack already reads: RLDS, LeRobot, WebDataset, HDF5, zarr or Rerun .rrd, or raw streams with metadata. Readable converter source ships alongside the data so the schema can be retargeted to your own loader.

From signed spec to first validated batch, the timeline is a median of 48 hours where coverage is deep and three to four weeks where it is thin and contributors need to be recruited first. After the first batch, on-demand top-ups run against the same accept criteria, so you can add more attempts or more locations without rewriting the spec.

Describe the task, the locations it moves through, its sub-steps and their order, and the interruptions or variation your pipeline is missing. That becomes a written skill spec, quoted for reach, timeline and price before capture begins, and most programs run a Spec Pilot first.