What Is the Universal Manipulation Interface (UMI)?

UMI stands for Universal Manipulation Interface. It is described in the paper Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots and on its project site, and the authors open-source it. The paper frames a choice between teleoperated robot data, which needs costly hardware and expert operators, and in-the-wild human video, which has a large embodiment gap to robots. UMI sits between them: a sensorized handheld gripper that people carry into any environment, so no robot is needed during collection.

In UMI the demonstrator holds a gripper, so every recorded motion is a gripper motion. In egocentric human video the contributor uses their own hands, with pose and action layers delivered beside the video. Firsthand is not part of UMI, endorsed by it or related to its authors or their institutions. The site states no default compatibility and does not state that Firsthand builds or collects with a handheld gripper. It states egocentric human capture: head and wrist cameras worn by paid, consented contributors, collected fast, on demand and to a written skill spec, under a perpetual, buyer-owned commercial license. Gripper width and robot-frame end-effector actions are not stated as delivered fields.

The paper is by Cheng Chi, Zhenjia Xu and co-authors, with affiliations at Stanford University, Columbia University and Toyota Research Institute. The project page lists it at Robotics: Science and Systems (RSS) 2024.

The stated motivation is that earlier handheld devices struggled to balance action diversity with transferability, because of limited visual context, imprecise actions, latency differences and weak policy representations.

The hardware is a trigger-activated, handheld, 3D-printed parallel-jaw gripper with soft fingers. A GoPro is mounted on it as the only sensor and recording device, with a 155-degree fisheye lens and two physical side mirrors that give an implicit stereo view. A second gripper can be added for bimanual tasks.

The paper describes a four-step protocol: optional clock synchronisation, optional gripper calibration, a mapping scan of each new scene (usually around one minute), then demonstrations recorded as one video each.

Each demonstration is one mp4 from the gripper-mounted camera, with the camera’s built-in IMU data stored in the file. Actions are recovered afterwards: an inertial-monocular SLAM system built on ORB-SLAM3 estimates a 6-DoF end-effector pose at metric scale, and fiducial markers track finger width. A script matches left and right grippers by camera serial number and converts the videos into a training dataset.

The GitHub repository shows that pipeline writing a dataset.zarr.zip file. Against a motion-capture benchmark, the paper reports a mean absolute trajectory error of 6.1 mm and 3.5 degrees, and lists the need for enough scene texture as a limitation.

The policy takes a sequence of synchronised observations (RGB image, relative end-effector pose and gripper width) and outputs relative end-effector poses and gripper width. The paper uses Diffusion Policy for all experiments, with inference-time latency matching and a relative-trajectory action representation, and describes the resulting policies as hardware-agnostic.

Because the camera is fixed to the fingers, the paper says mounting the setup on a robot needs no camera-robot-world calibration.

The authors evaluated four tasks: cup arrangement, dynamic tossing, bimanual cloth folding and dish washing. In an in-the-wild experiment, 3 demonstrators collected 1,400 cup-arrangement demonstrations in 12 person-hours across 30 locations. The policy reached 71.7% success (43 of 60 trials) in two unseen environments, while a policy trained only on narrow-domain data scored 0%.

On cup arrangement, the gripper was more than 3 times faster than space-mouse teleoperation and ran at 48% of bare-hand speed. These are the authors’ results on their own setup; Firsthand makes no performance claim against them.

Write the target skills, environments and signals into a skill spec and confirm them on a Spec Pilot, a small fast first batch. This is on-demand custom collection of egocentric human data, not a gripper program and not a catalog download.

Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks. Billing is per validated hour.

What is the Universal Manipulation Interface?

An open-source framework that collects robot demonstrations with a handheld parallel-jaw gripper carrying a GoPro, then trains robot policies from them, per its paper.

How does UMI collect handheld gripper data?

A person records each demonstration as one mp4. SLAM and fiducial markers then recover end-effector pose and finger width.

Is UMI egocentric human video?

No. The camera is mounted on a gripper, so the recorded motion is a gripper motion rather than a human hand.

Does Firsthand use UMI or a handheld gripper?

The site does not state that it does. Firsthand is not part of UMI or endorsed by it, and the site states no default compatibility.

What licence does UMI use?

The GitHub repository is released under the MIT licence. The hardware design licence is Not stated in the sources read.

How fast can custom pretraining data be collected?

Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks.