What Is Vid2Robot?

Vid2Robot is named in Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformers (arXiv 2403.12943) by Vidhi Jain, Maria Attarian, Nikhil J Joshi, Ayzaan Wahid, Danny Driess, Quan Vuong, Pannag R Sanketi, Pierre Sermanet, Stefan Welker, Christine Chan, Igor Gilitschenski, Yonatan Bisk and Debidatta Dwibedi, from Google DeepMind, Carnegie Mellon University and the University of Toronto, submitted on 19 March 2024 and listed at Robotics: Science and Systems 2024. The authors note that large multi-task manipulation systems usually specify the task with text, and ask whether a robot can instead learn by observing a person despite differences in embodiment and environment.

Firsthand is not affiliated with the Vid2Robot authors and makes no claim that any model, Vid2Robot included, was trained on Firsthand data. Firsthand captures real human egocentric video, custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license, with one task instruction per episode. Paired human-robot video, robot trajectories, teleoperation data, generated or synthetic video, synthetic data, point tracks, trajectory labels and language annotations beyond that instruction are not Firsthand deliverables.

Per the paper, Vid2Robot is an end-to-end video-conditioned policy. The task is not given as text or as a goal image: a prompt video of a person doing the task is the specification, and the policy produces the robot actions that perform the same task from the robot’s own camera view. The authors argue that a verb such as open can mean very different motions for a drawer, a cabinet or a jar, and that a video can disambiguate what text leaves open. They state that language is not an input to Vid2Robot.

The policy has four modules. A prompt video encoder and a robot state encoder each use a per-frame image encoder followed by a Perceiver Resampler. A state-prompt encoder then lets the state tokens act as queries over the prompt tokens as keys and values, so the model learns which object in the current scene the prompt refers to. A robot action decoder cross-attends to those prompt-aware state tokens to predict discretized actions in one forward pass.

The authors give the reason for cross-attention as cost: self-attention over all video tokens would be far larger, and resampling to a small set of latent tokens makes training on paired videos practical.

The main objective is behavior cloning, a cross-entropy loss over discretized actions. The authors add three auxiliary losses to encourage representations that ignore embodiment, lighting and background: a temporal video alignment loss, a contrastive loss between a prompt video and a robot video of the same task, and a contrastive loss between videos and the task text. Their ablation reports that the auxiliary losses help, and that the video-text loss adds little beyond the other two.

Per the paper, training needs pairs of a prompt video and a robot trajectory that perform the same task. The authors build the pairs from three sources: existing robot-robot videos of the same task, hindsight human-robot pairs in which people perform tasks from the robot’s viewpoint to match existing robot trajectories, and a small share of co-located human-robot pairs recorded in the same workspace. They describe the mix as roughly 100k robot videos and 10k human videos, more than 90% of it robot-robot, with task instructions used only to pair videos.

These counts are the authors’ descriptions, and Firsthand has not verified them. Robot trajectories are a required input, which is a different thing from human capture alone.

The authors evaluate on a mobile manipulator, the Google Robot, against BC-Z, a video-conditioned baseline. With human prompt videos, their Table I shows 52.8% overall success for Vid2Robot against 30.6% for BC-Z, the over 20% improvement stated in the abstract; with robot prompt videos the two are close at 54.9% and 52.6%. For cross-object motion transfer they report 34.2% against 17.5%. These are the authors’ figures on their robot and tasks, and Firsthand has not verified or reproduced them.

Per the paper, failures come from self-occlusion, grasping errors with small, transparent or deformable objects, and distractor objects or lighting causing the wrong object to be chosen. The demonstrations were short task clips of 5 to 20 seconds, and the authors note that long-horizon or in-the-wild videos would need effective pairing strategies. They keep the robot base fixed in their experiments. They state that they will release code and checkpoints; no official repository is confirmed from the paper or project page.

The first table sets Vid2Robot beside three site entries. The Gen2Act paper cites Vid2Robot as a baseline and builds on the idea of conditioning a policy on a human video, but Gen2Act generates that video rather than recording it. The Vid2Robot paper names RT-1 and RT-2 as language-conditioned systems and positions video prompts as complementary to language, goal images and other task specifications. Its comparisons are the authors’ own.

Per the paper, the method needs prompt videos of people performing tasks and robot trajectories for the same tasks, plus language instructions to pair them. The robot side is data a buyer would have to supply. The paper does not state whether egocentric capture suits the prompt side, so that is Not stated.

Firsthand states real human egocentric capture with one task instruction per episode and depth derived from a calibrated stereo rig. That is a note about format, not a claim of compatibility, and Firsthand does not supply paired human-robot video or robot trajectories.

When a program needs real human demonstrations that public or synthetic data does not cover, specified field by field. Firsthand does not build or train models. Confirm the fields on a Spec Pilot, a small fast first batch; the median from signed spec to first validated batch is 48 hours where coverage is deep, three to four weeks where thin.

What is Vid2Robot?

A published framework (arXiv 2403.12943) in which a human demonstration video is the task prompt and an end-to-end policy outputs robot actions. It is third-party research.

How does Vid2Robot use cross-attention?

Per the authors, state tokens from the robot’s current view act as queries over prompt video tokens, and an action decoder cross-attends to the result to predict actions. It also keeps attention cost small.

What results do the authors report?

In their Table I, 52.8% overall success with human prompt videos against 30.6% for BC-Z. These are the authors’ figures; Firsthand has not verified them.

How does Vid2Robot differ from Gen2Act?

Vid2Robot takes a recorded demonstration video as the prompt. Gen2Act, per its authors, generates a human video from a scene image and a language task and conditions the policy on that. The Gen2Act paper uses Vid2Robot as a baseline.

Does Firsthand use Vid2Robot, and can I request a custom dataset?

Firsthand is not affiliated with the Vid2Robot authors and claims no use, training or compatibility. Paired human-robot video, robot trajectories and teleoperation data are not Firsthand deliverables. You can request a custom dataset.