What Is WHIRL (Human-to-Robot Imitation in the Wild)?

WHIRL is named in Human-to-Robot Imitation in the Wild (arXiv 2207.09450) by Shikhar Bahl, Abhinav Gupta and Deepak Pathak of Carnegie Mellon University, submitted on 19 July 2022 and published at RSS 2022 according to the arXiv comments. The authors expand the name as In-the-Wild Human Imitating Robot Learning. They argue that common imitation learning relies on many kinesthetic or teleoperated demonstrations per task, which can be expensive and may not generalize to new settings or robots, and that most robot manipulation experiments stay in simulation or in the lab. Their alternative is visual imitation: the robot observes a person, with pixels only and no access to the person’s actions. They name three challenges: the embodiment mismatch between human and robot bodies, the lack of human actions, and the lack of task information such as rewards beyond pixels. For the broader idea, see the site’s imitation learning and embodiment gap entries.

Firsthand is not affiliated with the WHIRL authors and makes no claim that any model, WHIRL included, was trained on Firsthand data. WHIRL learns from a third-person video; Firsthand does not capture third-person or fixed external-camera footage. Firsthand captures real human egocentric video, custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license, with one task instruction per episode. Hand-pose retargeting outputs, robot trajectories, action labels, reward labels, pretrained representations, embeddings, teleoperation data, paired human-robot video, synthetic data and language annotations beyond that instruction are not Firsthand deliverables.

The authors describe three components: watch, repeat and improve. The robot first watches a human video and obtains priors such as hand movement and object interactions. It repeats those priors by interacting in the real world, both trying to achieve the task and exploring around the prior. It then improves its task policy with an agent-agnostic objective that aligns human and robot videos. The abstract lists three key contributions: a simple sampling-based policy optimization approach, a novel objective function for aligning human and robot videos, and an exploration method to boost sample efficiency.

Per the paper, each frame is passed through the 100DOH hand-object detector, which gives a hand box and a discrete contact variable, and the MANO hand parameterization supplies the rotation of the wrist. Because out-of-the-box detections in unstructured settings are noisy, the contact signal is smoothed with a Savitzky–Golay filter, which gives the start and end of the interaction. The prior is a small set of interaction waypoints, a wrist orientation measure and open or close commands for the hand. A depth image from an external camera projects these into the robot’s frame. The authors note that the prior alone does not usually succeed, because of morphology differences, detection errors and calibration errors.

The task policy outputs a residual to the prior, which the authors say keeps early behaviour close to the prior and therefore likely safe. They fit it as a conditional variational auto-encoder so it can capture several ways of doing a task. A separate exploration policy aims to maximize the change the robot causes in the environment. Optimization is zeroth-order and sampling-based, similar to the Cross Entropy Method: candidate trajectories are run on the robot, ranked by cost, and the best ones refit the policy.

To compare a human video with a robot video despite different bodies, the authors inpaint the agent out of both videos with Copy-and-Paste Networks and compare action-recognition embeddings of the results. They argue the effect on the environment matters more than how the agent moved.

The authors report experiments on 20 manipulation tasks across 3 environments, using a Hello Robot Stretch mobile manipulator and an Intel RealSense D415 depth camera, in everyday kitchen-style settings. On the drawer task they report training success rising from about 43% to 83%, and on the door task from about 40% to 92%. These are the authors’ figures on their setup; Firsthand has not verified them.

They also report limits. A shelf pick-and-place task needed more precision, and waypoints off by even a couple of centimetres could make the robot hit the shelf. Performance in new scenes, where calibration and geometry change, was generally lower than on training instances. The conclusion lists learning directly from passive offline video, without online interaction or per-task demonstrations, as future work.

The first table sets WHIRL beside four site entries, summarising what those entries state. The main difference is that WHIRL starts from one human demonstration per task and then spends real robot interaction to close the gap, while the other methods pre-train on larger sets of human video.

Per the paper, it needs a human video of the task, a hand-object detector, hand and wrist pose estimation, an external depth camera and calibration to project into the robot frame, and a robot to run trials. The project page also lists a paired human-to-robot dataset of videos for the 20 tasks, produced by running WHIRL; that is the authors’ dataset, and paired human-robot video is not a Firsthand deliverable.

Firsthand states real human egocentric capture, depth from a calibrated stereo rig, 3D hand pose, and one task instruction per episode. That is a note about format, not a compatibility claim: WHIRL uses a third-person view, any waypoints or contact variables would come from tools the buyer runs, and the robot trials would be the buyer’s own. No off-the-shelf dataset for these tasks is stated here, so this is custom collection: request on-demand collection and confirm it on a Spec Pilot, with a median of 48 hours to a first validated batch where coverage is deep and three to four weeks where it is thin.

What is WHIRL?

WHIRL, In-the-Wild Human Imitating Robot Learning, is a published method (arXiv 2207.09450, RSS 2022) in which a robot learns a manipulation task from one third-person human video and then improves through real-world trials. It is third-party research.

What does WHIRL take as input?

A human video of the task. Per the paper, hand and contact detections from that video become a prior over waypoints, wrist orientation and open or close commands, which a depth camera projects into the robot frame.

Does WHIRL need robot demonstrations or teleoperation?

The authors state they use imitation data only as a prior and do not use teleoperated or kinesthetic demonstrations. The robot instead improves through its own real-world trials, scored by an agent-agnostic comparison with the human video.

What did the authors report for WHIRL?

They report experiments on 20 manipulation tasks across 3 environments, with training success on a drawer task rising from about 43% to 83%. These are the authors’ figures; Firsthand has not verified them.

Was WHIRL trained on Firsthand data?

No. Firsthand is not affiliated with the authors and claims no use, training or compatibility. Firsthand does not capture third-person footage, and robot trajectories or paired human-robot video are not Firsthand deliverables.

How can I request a custom egocentric dataset for human-to-robot learning?

Request a custom dataset with your objects, interactions and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep.