What Is Dreamitate?

Dreamitate is named in Dreamitate: Real-World Visuomotor Policy Learning via Video Generation (arXiv 2406.16862) by Junbang Liang, Ruoshi Liu, Ege Ozguroglu, Sruthi Sudhakar, Achal Dave, Pavel Tokmakov, Shuran Song and Carl Vondrick, from Columbia University, Toyota Research Institute and Stanford University, submitted on 24 June 2024. The official repository describes it as a CoRL 2024 paper. The authors say the name means dream, then imitate. They want a visuomotor policy that generalizes to diverse visual environments, and they note that behavior cloning needs ground-truth robot actions, which is hard to scale.

Firsthand is not affiliated with the Dreamitate authors and makes no claim that any model, Dreamitate included, was trained on Firsthand data. Firsthand captures real human egocentric video, custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license, with one task instruction per episode. Generated or synthetic video, tool or object pose labels, 6D tool trajectories, robot trajectories, teleoperation data, paired human-robot video, synthetic data and language annotations beyond that instruction are not Firsthand deliverables.

Per the paper, there are three steps. First, capture stereo recordings of a human using a tool to perform the task. Second, fine-tune a video model to recreate those demonstrations from an initial scene image. Third, in a new scene, give the model a stereo image pair, generate the manipulation video, track the tool in it and execute that trajectory on the robot. The authors call this synthesize-then-track, and say it avoids teleoperation. They use Stable Video Diffusion as the starting point and train a separate model for each task. See the site’s visuomotor policy entry for the broader term.

The authors’ key insight is that common tools bridge the embodiment gap between the human hand and the robot manipulator. Humans demonstrate with custom tools whose CAD models are known; the same tool is mounted on the robot arm. Actions are the 6D pose of the tool, tracked with MegaPose in the generated frames and resolved in 3D from the stereo pair. The authors say this makes the pipeline agent-independent, because no agent-specific action space is learned. See the embodiment gap entry for the general problem.

Against a Diffusion Policy baseline trained on the same data, the authors report higher success on rotation, scooping and sweeping, and higher mIoU and lower rotation error on Push-Shape; the second table lists their counts. They also report that on the rotation task the model stays stable with one third of the demonstrations while the baseline declines. These are the authors’ figures on their setup; Firsthand has not verified or reproduced them.

Per the paper, actions must be visually trackable, the approach can fail when the end-effector is heavily occluded, reliance on rigid tools limits fine-grained control, and video models are costly enough that real-time closed-loop control is infeasible without faster inference. The paper says code and data will be released; the official repository is linked in the sources.

The third table sets Dreamitate beside three site entries that also use human video to drive a robot. The columns for the other methods summarise what the site entries state, not new claims. Dreamitate’s reference list does not include any of the three.

Per the paper, it needs stereo video of humans performing a task with a known, trackable tool from fixed tabletop cameras. The paper does not state whether egocentric capture would work as the video source, so that is Not stated. Firsthand states real human egocentric capture, with depth derived from a calibrated stereo rig and one task instruction per episode. That is a note about format, not a claim of compatibility, and any tool tracking would come from tools the buyer runs.

When a program needs real human demonstrations that public or generated data does not cover, specified field by field. Firsthand does not build or train models. Confirm the fields on a Spec Pilot, a small fast first batch; the median from signed spec to first validated batch is 48 hours where coverage is deep, three to four weeks where thin.

What is Dreamitate?

A published visuomotor policy framework (arXiv 2406.16862) that fine-tunes a video diffusion model on human demonstrations with tools, generates a task video for a new scene, tracks the tool and has a robot execute that trajectory. It is third-party research.

What does generate-then-track mean?

The model first generates a video of the task, then a tracker follows the tool in that video and the 6D tool poses become robot actions. The authors call this synthesize-then-track.

What results do the authors report?

On rotation, scooping and sweeping the authors report 37, 34 and 37 successes out of 40 trials, against 22, 22 and 5 for Diffusion Policy, plus better Push-Shape mIoU. These are the authors’ figures; Firsthand has not verified them.

What data does Dreamitate train on?

Per the paper, stereo video of humans using trackable 3D-printed tools, recorded by two fixed cameras instead of teleoperation, to fine-tune Stable Video Diffusion for each task.

Does Firsthand use Dreamitate, and can I request a custom dataset?

Firsthand is not affiliated with the Dreamitate authors and claims no use, training or compatibility. Generated video, tool pose labels, robot trajectories and teleoperation data are not Firsthand deliverables. You can request a custom dataset.