Egocentric Human Video vs UMI Gripper Data

UMI stands for Universal Manipulation Interface, a data collection and policy learning framework described in a paper by Chi and colleagues (arXiv 2402.10329) and on the project site, umi-gripper.github.io. Its collection hardware is a trigger-activated, handheld, 3D-printed parallel-jaw gripper with soft fingers. A GoPro camera is mounted on it as the only sensor and recording device, with a 155-degree fisheye lens and two physical side mirrors that give an implicit stereo view. A second gripper can be added for bimanual tasks.

The action labels come from the recording itself. The GoPro records its built-in IMU into the mp4 file, and an inertial-monocular SLAM system built on ORB-SLAM3 recovers a 6-DoF end-effector pose at metric scale. Gripper width is tracked continuously with fiducial markers. The paper trains policies with Diffusion Policy on relative end-effector trajectories and describes the resulting policies as hardware-agnostic. The central idea is that people demonstrate without a robot present, so data can be collected in any environment.

This page describes UMI only from those two official sources, and where they are silent it says not stated. Firsthand claims no relationship with the UMI authors or any lab, and the site does not state that Firsthand builds, sells or collects with UMI or any handheld gripper. The comparison is between two approaches to the same goal, manipulation demonstrations without a robot in the loop, not between two Firsthand offers. See /glossary/embodiment-gap for the term used below.

The difference starts with what the demonstrator holds. In UMI the demonstrator moves a gripper, so every recorded motion is already a gripper motion: an end-effector path and a finger-width signal. In egocentric human video the contributor uses their own hands while wearing a head camera and two wrist cameras, so the recorded motion is human hand and wrist motion, with pose labels delivered beside the video.

That choice moves the embodiment gap. The UMI paper argues that human video exhibits a large embodiment gap to robots and that most video demonstrations lack explicit action information. It states that UMI data has a minimal embodiment gap in action and observation for robots fitted with the matching gripper and camera setup. The gap does not disappear for human video; it is addressed downstream, for example by retargeting (see /glossary/motion-retargeting). Firsthand delivers pose and action layers measured on the rig, not a retargeted robot action, and no retargeting step is stated as delivered.

The two methods also differ in who does the work. The UMI hardware is open-sourced, so a team builds or sources the grippers, trains demonstrators and runs scene mapping itself. Firsthand collects custom egocentric data on demand to your written skill spec and delivers validated episodes. Neither is better in the abstract. The useful question is which one matches the embodiment your policy will run on and the observations it consumes.

The first table sets them side by side. A UMI demonstration is one mp4 per demonstration from the gripper-mounted GoPro, with IMU data embedded. Actions are derived afterward: the paper describes a script that converts the mp4 files into a training dataset, recovering robot actions with the SLAM pipeline and fiducial marker detection. For each new scene the collection protocol first records a mapping video of about a minute, so every demonstration in that scene is localized to one map. How the converted dataset is laid out on disk is not stated here.

Firsthand records what the wearer sees and does: a global-shutter head camera at 4K and 60 fps, two wrist cameras at 1920×1080 and 60 fps, depth at 848×480 and 30 fps recovered from a calibrated stereo pair, and 6-DoF IMUs on the head and both wrists at 200 Hz. Pose and action layers ship with it, all on one PTP clock with a median skew of 1.12 ms. The structural difference is that UMI’s camera rides on the tool that will later ride on the robot, while Firsthand’s cameras ride on a person.

The second table maps each delivered field to a use and its field name. Hand pose is hand_joints, float32 [T, 2, 21, 3] in metres in the head-camera frame, and hand_visible flags occluded joints. Wrist motion is measured, not estimated from video: one 6-DoF IMU per wrist at 200 Hz. A wrist pose derivation, coordinate frame, accuracy and smoothing are set per program, so this page does not claim a metric wrist position path in a world frame. See /custom-collection/egocentric-wrist-trajectory-data and /custom-collection/egocentric-hand-tracking-data.

Both approaches rely on wrist-level views; see /compare/head-vs-wrist-cameras for the trade-offs.

Handheld gripper demonstrations are the better fit when your robot has a parallel-jaw gripper and you want the demonstrated action to be that gripper’s action. UMI records the end-effector pose and the finger width directly, so there is no hand-to-gripper mapping. The project site says the same policy can be deployed on any robot equipped with a parallel jaw with a stroke above 85 mm, and the paper shows it on a UR5 and a Franka arm for dynamic tossing, bimanual cloth folding and dish washing. Firsthand states none of this: it delivers no gripper width and no robot-frame end-effector action.

It is also the better fit when the camera view must match the robot’s. Because the camera is mechanically fixed to the fingers, the paper says that mounting the setup on a robot needs no camera-robot-world calibration, and that the videos seen by the wrist camera are almost indistinguishable between demonstration and deployment. If your policy will run on exactly that wrist view, gripper data supplies it.

Third, it is the better fit when you want to own the collection loop. The paper reports numbers for this route, attributed to UMI’s own experiments rather than to Firsthand: on a cup arrangement task the gripper was more than three times faster than space-mouse teleoperation and ran at 48% of human-hand speed, and its SLAM tracking showed a mean absolute trajectory error of 6.1 mm in position and 3.5 degrees in rotation against a motion-capture benchmark. Firsthand publishes no hand or body pose accuracy figure and states no throughput, so this page makes no like-for-like claim.

Finally, UMI has stated limits worth weighing. Its SLAM approach needs enough texture in the environment, and the paper reports that collecting with the gripper is less efficient than human hand demonstration, partly because of the gripper’s weight and bulk and its reduced degrees of freedom compared with a hand.

Egocentric human video scales better when the demonstrator should move naturally and the dataset needs to span many tasks and environments. A contributor wearing the rig performs the task with their own hands, so there is no device to build, carry or hold. The UMI paper itself reports that gripper collection ran slower than bare-hand demonstration; egocentric capture is bare-hand demonstration with sensing attached. The site states no throughput or hour count, and this page adds none; scale here means capture wherever the task is done.

It is also better when finger-level motion matters. A parallel-jaw gripper reduces a hand to a pose and one width. Firsthand delivers 21 joints per hand with a per-joint visibility flag, contact events marking when a hand meets an object, and action segments (verb, noun and nanosecond timestamps, from 97 verbs and 512 nouns). The EgoScale paper (arXiv 2602.16710) makes a related point: it notes that many prior approaches focus on grippers or low-DoF hands, where fine-grained finger articulation is absent, and it supervises on wrist motion and retargeted hand articulation instead.

Third, the labels are measured on the rig rather than reconstructed afterward. UMI derives actions after recording with SLAM, which the paper says needs sufficient texture in the environment. EgoScale likewise derives camera motion and hand pose with off-the-shelf SLAM and hand-pose estimation. Firsthand’s wrist motion is measured by IMUs at 200 Hz, and depth comes from a calibrated stereo pair that the site states stays accurate outdoors and indoors alike. How either approach behaves in texture-poor scenes is not stated for Firsthand’s pose layers.

See /custom-collection/time-synchronized-egocentric-data for the sync details and /custom-collection/action-labeled-egocentric-video for the label layers.

The site does not state a gripper width channel, a gripper state, or robot-frame end-effector action labels as delivered fields. The vision-language-action glossary entry mentions gripper state as one kind of action stream a VLA program needs, but the episode schema field list does not itemise one, so treat it as not stated as a delivered field and ask in writing if you need it. The site also does not claim a head or camera 6-DoF pose or a SLAM trajectory, which is exactly what UMI recovers as its action signal. A head IMU at 200 Hz is delivered; it is inertial data, not a camera pose.

There is also no third-person capture, no narration or dense captions, and no evaluation or benchmark datasets: collection is custom to your spec. The only language field is a task instruction paired to each episode. Firsthand states no hand or body pose accuracy figure, no relative end-effector trajectory representation, and no kinematic filtering against a particular robot. And the site does not state that it offers UMI or handheld-gripper collection.

The third table lists each gap. A “not stated” row means no page on this site says the item is delivered, not that it is impossible; raise any you need as a skill-spec question before the quote.

Yes, and the two answer different needs. A common split is breadth from human video and embodiment match from gripper data. Egocentric human data supplies many tasks, objects and environments with hand and action labels, and a smaller gripper set supplies end-effector actions and gripper widths in the form your target robot uses. The EgoScale paper follows a related pattern: it pretrains on large-scale egocentric human data, then adds a lightweight stage of aligned human-robot mid-training. That recipe uses its own robot and capture setup and is neither UMI data nor Firsthand data. See /compare/human-demonstrations-vs-teleoperation for the same kind of blend with teleoperation.

Practically, keep the sets separable. Record the source of each episode in your own training manifest, keep each set’s action representation distinct, and do not mix hand-pose labels with gripper-pose labels in one action space without a deliberate mapping. Firsthand does not deliver the gripper side and does not state an alignment method with an outside system. Agree shared events, such as a start and stop cue, in the skill spec, and align using the nanosecond timestamps each stream carries.

Firsthand claims no relationship with the UMI authors, with any lab cited here, or with any gripper hardware supplier.

Send the tasks, the environments and regions, and the delivery format you need, and name the pose layers in the streams and labels list. RGB is the baseline and the other streams are added where the spec calls for them. If egocentric data is meant to complement gripper data, also say which action signal your model consumes and which gripper-side signals you will source elsewhere. Hand pose, segmentation and action labels each add equipment, QA passes and cost, so order what the model consumes. The order flow is spec, quote, Spec Pilot, then production, and there is no list price: billing is per validated hour.

For formats, the episode schema is the canonical layout and HDF5 and zarr mirror it. In the LeRobot v2.1 export, hand and body pose are flattened into one observation.state vector, with 2×21×3 hand joints first and then 24×3 body joints, so keep that order when you slice it. RLDS, WebDataset and Rerun .rrd layouts for pose are not itemised on the site, so name your loader in the spec.

For quality, every episode is checked on sync within the ceiling, calibration, coverage and consent, and a rejected episode is recaptured at the provider’s cost. Faces, plates and screens are detected, tracked and blurred before anything leaves QA, and every contributor signs a documented release. Delivered data ships under a perpetual, buyer-owned license. The last table lists the spec items and what the site states for each.