What Is OKAMI (Humanoid Manipulation from Single Video Imitation)?

OKAMI is named in OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitation (arXiv 2410.11792) by Jinhan Li, Yifeng Zhu, Yuqi Xie, Zhenyu Jiang, Mingyo Seo, Georgios Pavlakos and Yuke Zhu, accepted for oral presentation at the 8th Annual Conference on Robot Learning. The name expands to Object-aware Kinematic retArgeting for huManoid Imitation. The authors say deep imitation learning for humanoid manipulation relies on large amounts of whole-body teleoperation, which demands domain expertise and strenuous effort, and they ask whether a humanoid can instead acquire a skill by watching a human perform it once. For the broader ideas, see the site’s imitation learning and motion retargeting entries.

Firsthand is not affiliated with the OKAMI authors and makes no claim that any model, OKAMI included, was trained on Firsthand data. OKAMI is evaluated on a Fourier GR1 humanoid; Firsthand does not claim to use that hardware. Firsthand captures real human egocentric video, custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license, with one task instruction per episode. Object-aware retargeting outputs, reference plans, SMPL-H body reconstructions, robot trajectories, action labels, reward labels, pretrained representations, embeddings, teleoperation data, paired human-robot video, synthetic data and language annotations beyond that instruction are not Firsthand deliverables.

Per the paper, OKAMI is a two-stage method. Stage one builds a reference plan from a single RGB-D video: GPT-4V names the task-relevant objects, Grounded-SAM and the Cutie video segmenter localise and track them, the authors’ extension of SLAHMR reconstructs body and hand motion as SMPL-H models using HaMeR hand estimates, and CoTracker keypoint velocities mark the keyframes that split the video into subgoals. Each plan step holds the point clouds of a target object and a reference object plus a segment of the human motion.

Stage two is object-aware retargeting. The system localises the same objects in the robot’s view, maps the body motion into the humanoid’s task space, warps the arm trajectory to the new object locations and solves inverse kinematics for the arms. Hand poses are retargeted to the robot’s fingers separately. The authors say this factorised design lets one video adapt to different object positions, backgrounds and new instances of the same object types.

The paper makes two assumptions about the video: every frame captures the human body, and the camera is static throughout the recording. It also requires depth. The authors list the RGB-D requirement as a limit, because it rules out in-the-wild Internet videos recorded in RGB, and name extending OKAMI to web videos as future work. They also report that a demonstrator who moved faster produced noisier human-model reconstruction. These conditions describe a fixed-camera recording of a person, not a head-worn first-person view; that comparison is Firsthand’s reading, not the authors’.

The authors evaluate six tasks (plush toy in basket, sprinkle salt, close the drawer, close the laptop, place snacks on plate and bagging) on a Fourier GR1 humanoid with two 6-DoF Inspire dexterous hands and a RealSense D435i camera, running 12 trials per task. They report a 71.7% task success rate averaged across tasks, 58.3% above the ORION baseline. They then train closed-loop visuomotor policies with the ACT behavioral cloning algorithm on OKAMI’s successful rollouts for sprinkle salt and bagging, reporting an average 79.2% success without teleoperation, with policies improving as more rollouts were collected. These are the authors’ figures on their setup; Firsthand has not verified them.

The authors also list limits. The work focuses on upper-body retargeting for tabletop workspaces, with lower-body locomotion left for future work. It needs RGB-D video, and retargeting has limited robustness against large variations in object shape. In simulation they attribute much of the gap to a version given ground-truth object poses to perception noise and partial object point clouds.

The first comparison table sets OKAMI beside six site entries, summarising what those entries state. The main difference is the unit of supervision: OKAMI starts from one video per task and generates robot rollouts, whereas HAT pairs a dataset with a policy, EgoZero learns from smart-glasses demonstrations and the zero-shot entry learns from diverse web video.

Per the paper, it needs a recorded RGB-D video of a human performing the task with a static camera and the body in frame, open-world vision models, a human-motion reconstruction pipeline and a humanoid with dexterous hands.

Firsthand states real human egocentric capture, depth from a calibrated stereo rig, 3D hand pose at 21 joints per hand, and one task instruction per episode. That is a note about format, not a compatibility claim: OKAMI’s inputs are fixed-camera RGB-D videos, so Firsthand’s egocentric footage belongs to the site’s general capture offer, not to this method. Plan generation, retargeting and robot rollouts would come from tools the buyer runs. No off-the-shelf dataset for this is stated here, so this is custom collection: request on-demand collection and confirm it on a Spec Pilot, with a median of 48 hours to a first validated batch where coverage is deep and three to four weeks where it is thin.

What is OKAMI?

OKAMI is a published method (arXiv 2410.11792) that generates a manipulation plan from a single RGB-D human video and uses object-aware retargeting to have a humanoid robot imitate it. It is third-party research.

What hardware does OKAMI use?

Per the paper, a Fourier GR1 humanoid with two 6-DoF Inspire dexterous hands and a RealSense D435i camera. Firsthand does not claim to use that hardware.

Does OKAMI need teleoperation data?

No. The authors state it needs only a single RGB-D human video per task, and that policies trained on OKAMI rollouts avoid labor-intensive teleoperation.

What did the authors report for OKAMI?

They report a 71.7% average task success rate over six tasks and a 79.2% average for policies trained on OKAMI rollouts. These are the authors’ figures; Firsthand has not verified them.

Was OKAMI trained on Firsthand data?

No. Firsthand is not affiliated with the authors and claims no use, training or compatibility. Reference plans, retargeting outputs, robot trajectories, action labels and paired human-robot video are not Firsthand deliverables.

How can I request a custom egocentric dataset for human-to-robot learning?

Request a custom dataset with your objects, interactions and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep.