What Is MimicPlay?

MimicPlay is named in MimicPlay: Long-Horizon Imitation Learning by Watching Human Play (arXiv 2302.12422) by Chen Wang, Linxi Fan, Jiankai Sun, Ruohan Zhang, Li Fei-Fei, Danfei Xu, Yuke Zhu and Anima Anandkumar. The project page lists Stanford, NVIDIA, Georgia Tech, UT Austin and Caltech, and the arXiv record notes a CoRL 2023 oral presentation. It was submitted on 24 February 2023 and last revised on 13 October 2023. The authors start from a data problem: learning complex long-horizon tasks from human demonstrations often requires an unattainable amount of demonstrations. Their answer is to lean on cheaper human play data, and they note that a human completes a long-horizon task much faster than a teleoperated robot.

Firsthand is not affiliated with the MimicPlay authors and makes no claim that any model, MimicPlay included, was trained on Firsthand data. Firsthand captures real human egocentric video, custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license, with one task instruction per episode. Free-form play data, hand-trajectory labels, latent plans, robot trajectories, teleoperation data, paired human-robot video, synthetic data and language annotations beyond that instruction are not Firsthand deliverables.

Per the project page, training has two stages. In stage one, cheap human play data trains a goal-conditioned trajectory generation model, which builds a latent plan space holding high-level guidance for diverse task goals. The conclusion says this planner learns by predicting future 3D human hand trajectories given a goal image. In stage two, a small amount of multi-task teleoperated robot data trains a low-level controller conditioned on latent plans from the frozen planner. See the site’s imitation learning entry for the broader term.

The authors define human play data as video sequences of people freely interacting with the environment using their hands. They hypothesize that, even with different morphologies, it holds rich and salient information about physical interactions that can facilitate robot policy learning. Play video is not tied to one task or one robot, which is why the authors call it cost-effective and pair it with a small set of teleoperated demonstrations for the robot-specific part. How much play data they used is not stated in the sources read for this entry.

Per the project page, MimicPlay takes a single long-horizon task video prompt, either a human motion video or a robot teleoperation video. It generates latent plans from that prompt and guides the low-level controller through the task. The authors present this as an intuitive interface for specifying goals with human videos, and show the same trained model being prompted by human motion. They also report that, after multi-task training in one model, it generalizes one-shot to new tasks with unseen temporal compositions. The site’s long-horizon task entry explains why ordered sub-steps matter for this kind of data.

The authors evaluate on 14 long-horizon manipulation tasks in the real world and say MimicPlay outperforms state-of-the-art imitation learning methods in task success rate, generalization ability and robustness to disturbances. The project page conclusion states it outperforms prior work by over 50 percent on those tasks. It also shows real-time re-planning after a human disturbs the scene. These are the authors’ claims on their setup; Firsthand has not verified or reproduced them, and the per-task counts are Not stated here.

The second table sets MimicPlay beside three site entries that also use human video for robot manipulation. The columns for the other methods summarise what those entries state, not new claims. MimicPlay’s own sources read here do not discuss the other three, so no head-to-head comparison is made.

Per the sources read, it needs two things: human play video, and a small set of teleoperated robot demonstrations. The authors describe hand trajectories in 3D as the planner’s prediction target, but the sources read do not state the capture hardware or format for the play data, so that is Not stated. Firsthand states real human egocentric capture with stereo depth, 3D hand pose and, for long-horizon tasks, ordered action segments inside each episode. That is a note about format, not a claim of compatibility, and any trajectory targets would be derived by the buyer.

When a program needs real human demonstrations that public data does not cover, specified field by field, including multi-step tasks. Firsthand does not build or train models, or supply robot data. Confirm the fields on a Spec Pilot, a small fast first batch; the median from signed spec to first validated batch is 48 hours where coverage is deep, three to four weeks where thin.

What is MimicPlay?

A published hierarchical imitation learning framework (arXiv 2302.12422) that learns latent plans from human play video to guide a low-level robot controller trained on a small number of teleoperated demonstrations. It is third-party research.

What is human play data?

Per the authors, video sequences of people freely interacting with the environment using their hands. They treat it as a cheaper source of information about physical interactions than robot demonstrations.

What results do the authors report?

The authors report evaluation on 14 real-world long-horizon manipulation tasks and say MimicPlay outperforms prior imitation learning methods in success rate, generalization and robustness to disturbances. These are their claims; Firsthand has not verified them.

How is MimicPlay different from EgoMimic?

Per the site entries, EgoMimic co-trains one policy on egocentric human data and robot data. MimicPlay splits the problem: a planner learned from human play video and a controller trained on teleoperated demonstrations.

Does Firsthand use MimicPlay, and can I request a custom dataset?

Firsthand is not affiliated with the MimicPlay authors and claims no use, training or compatibility. Play data, latent plans, trajectory labels and teleoperation data are not Firsthand deliverables. You can request a custom dataset.