Egocentric Video vs Motion Capture for Robot Training Data

Motion capture is a family of methods that measure human body movement inside an instrumented space. Marker-based systems track markers attached to the body with cameras placed around a capture volume; markerless systems estimate pose from several fixed cameras in the same kind of volume. Both are typically run in a studio or lab, and both observe the person from the outside. This page describes motion capture only as a general category. It names no vendor and states no accuracy, frame rate, marker count or price for any system.

Egocentric video is recorded from the person’s own viewpoint. Firsthand collects it custom, on demand, with contributors wearing a head camera and two wrist cameras while they perform your task in the real environment where it happens. Pose arrives as labels beside the video: 3D hand pose, a 24-joint body skeleton, wrist IMU streams, depth, contact events and action segments, all on one PTP clock.

The practical question is therefore not which method is more precise in the abstract. It is which one produces the observations and labels a robot policy will train on. Motion capture gives an external measurement of the body. Egocentric capture gives the kind of view a robot will have of the task, with pose attached. See /glossary/egocentric-video and /glossary/body-pose-estimation for the definitions.

Generic motion capture reports the position and orientation of tracked body segments over time, usually in a fixed world frame defined by the capture volume. Because the cameras sit outside the person, the whole body is observable, including torso, legs and head. What it does not record by itself is the person’s view: there is no first-person image stream, and the objects being handled appear only if the studio is arranged to record them. Depth, video or contact streams are separate systems that would have to be synchronized to it.

Firsthand’s capture records what the wearer sees and does: a global-shutter head camera at 4K and 60 fps, two wrist cameras at 1920×1080 and 60 fps, depth at 848×480 and 30 fps, and 6-DoF IMUs on the head and both wrists at 200 Hz. Hand and body pose are delivered with it as labels, so you do not have to infer them from the footage yourself. The trade-off is structural: the camera is worn by the person, so it cannot frame the wearer’s whole body.

The first table sets the two side by side across setting, viewpoint, signals, sync, scale, formats and license. Motion capture cells are general-category descriptions only; where a figure would depend on a specific system, the table says not stated rather than guessing.

Some delivered fields cover what teams often use motion capture for, and some add what a capture volume does not record. The closest equivalent to a body-motion trace is body_joints, a 24-joint 3D skeleton in the annotations group (bodyJoints[24][3]). Fine hand motion is covered by hand_joints, float32 [T, 2, 21, 3] in metres in the head-camera frame, with hand_visible flagging occluded joints. The site gives two frame descriptions for body pose, head-camera frame in the glossary and root-relative and world-frame in the stream list, so confirm the frame in the spec rather than assuming it.

Wrist motion is measured, not estimated from video: one 6-DoF IMU per wrist at 200 Hz, delivered as streams/imu_wrist_l and streams/imu_wrist_r. A wrist pose derivation, coordinate frame, accuracy and smoothing are set per program, so this page does not claim a metric wrist position path in a world frame. See /custom-collection/egocentric-wrist-trajectory-data for what is and is not stated.

What a body trace never carries is what the person was doing. Action segments (verb, noun and nanosecond timestamps, from 97 verbs and 512 nouns) and contact events mark the activity and the moment a hand meets an object. Depth, recovered from the calibrated stereo pair, adds scene geometry. The second table maps each field to a robot-learning use and its field name, and /custom-collection/egocentric-body-pose-data covers the pose layers in more depth.

Motion capture is the better fit when the requirement is a controlled, repeatable measurement of the whole body from outside. Full-body kinematics in a fixed world frame, including legs, torso and head, is what an instrumented volume is built for. If your question is biomechanical, or you need the body observed from a third-person viewpoint, a studio setup answers it directly and Firsthand does not: it offers no third-person capture and no camera or head 6-DoF pose.

It is also the better choice when the task can be performed inside the volume and you need a reference measurement. Teams commonly want an external ground truth against which to check a vision-based pose estimate, and a mocap session is a natural way to get one. If your requirement is a stated tolerance on joint position, check that tolerance against the specific system you are considering; Firsthand publishes no hand or body pose accuracy figure, so none is claimed here.

Finally, a studio gives you control over lighting, objects and the setting. That control is the reverse of what egocentric capture is for. When the goal is to isolate one variable under repeatable conditions, controlled capture is a reasonable choice, and this page does not argue otherwise.

Egocentric capture scales to the places and tasks you cannot move into a studio. Contributors perform the task where it actually happens, wearing the rig, to a written skill spec. Depth is recovered from the calibrated stereo pair rather than a separate active sensor, which the site states keeps it accurate outdoors and indoors alike. The scale here means capture can happen wherever the task is done; the site states no throughput or hour count for this offer, and this page adds none.

It also matches the observation a robot policy sees. The view comes from a head camera and two wrist cameras, so the footage resembles a robot’s own cameras more closely than an external view does. Pose and action labels arrive already aligned to that footage on one PTP clock, with a median skew of 1.12 ms, so the supervision and the pixels share a timeline. See /custom-collection/time-synchronized-egocentric-data for the sync details.

The labels are measured live on the rig rather than reconstructed afterwards. For contrast, the EgoScale paper (arXiv 2602.16710) derives its action labels from off-the-shelf SLAM and hand-pose estimation applied after capture. That is a different route from live capture. How either approach compares with motion capture on accuracy is not stated here. And because collection is on demand, the median time from a signed spec to a first validated batch is 48 hours where coverage is deep, and three to four weeks where it is thin.

The site does not claim a head or camera 6-DoF pose, and it does not claim a SLAM trajectory. A head IMU at 200 Hz is delivered, but that is inertial data, not a camera pose, and this page does not present it as one. There is no third-person capture, no narration or dense captions, and no evaluation or benchmark datasets: collection is custom to your spec. The only language field the site states is a task instruction paired to each episode.

Several pose details are also not stated and are set per program. No hand or body pose accuracy figure is published, the skeleton model and joint names are not stated, the body pose visibility flag is stated for hands only, and the frame rate of the body_joints stream is not stated. If your model depends on any of these, ask for it in writing before the quote.

The third table lists each gap with the status the site gives it. Treat every “not stated” row as exactly that: it is not a claim that the item is impossible, only that no page on this site says it is delivered. If you need one of them, raise it as a skill-spec question.

Yes, and the two answer different questions, so combining them is a design choice rather than a compromise. A common split is to use motion capture for a small controlled set of whole-body measurements, and egocentric capture for the breadth of tasks, objects and environments a policy has to see. The egocentric delivery supplies the first-person observation and the pose and action labels; the mocap session supplies an external reference.

One practical use is validation. A team can run the same task in a studio and in an egocentric collection, then compare how the delivered hand and body pose behave against the external measurement. Firsthand does not supply the mocap side, and the site does not state an alignment method between its streams and an outside system. Agree shared events, such as a start and stop cue, in the skill spec, and align using the nanosecond timestamps each stream carries.

Another use is priors. Whole-body motion measured in a volume can inform stance and reach assumptions, while egocentric data teaches the policy what the task looks like from its own cameras. Firsthand claims no relationship with any motion capture vendor or with the research groups cited below; the comparison here is between general approaches.

Send the tasks, the environments and regions, and the delivery format you need, and name the pose layers in the streams and labels list. RGB is the baseline and the other streams are added where the spec calls for them. Hand pose, body pose, segmentation and action labels each add equipment, QA passes and cost, so order what the model consumes. The order flow is spec, quote, Spec Pilot, then production, and there is no list price: billing is per validated hour.

For formats, the episode schema is the canonical layout and HDF5 and zarr mirror it. In the LeRobot v2.1 export, hand and body pose are flattened into one observation.state vector, with 2×21×3 hand joints first and then 24×3 body joints, so keep that order when you slice it. RLDS, WebDataset and Rerun .rrd layouts for body pose are not itemised on the site, so name your loader in the spec.

For quality, every episode is checked on sync within the ceiling, calibration, coverage and consent, and a rejected episode is recaptured at the provider’s cost. Faces, plates and screens are detected, tracked and blurred before anything leaves QA, and every contributor signs a documented release. Delivered data ships under a perpetual, buyer-owned license. The last table lists the spec items and what the site states for each.