What Is Gen2Act?

Gen2Act is named in Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulation (arXiv 2409.16283) by Homanga Bharadhwaj, Debidatta Dwibedi, Abhinav Gupta, Shubham Tulsiani, Carl Doersch, Ted Xiao, Dhruv Shah, Fei Xia, Dorsa Sadigh and Sean Kirmani, from Google DeepMind, Carnegie Mellon University and Stanford University, submitted on 24 September 2024. The official project page lists it at CoRL 2025. The authors ask how a manipulation policy can generalize to unseen object types and new motions without collecting robot data for every task, and answer by leaning on video generation models trained on web data.

Firsthand is not affiliated with the Gen2Act authors and makes no claim that any model, Gen2Act included, was trained on Firsthand data. Firsthand captures real human egocentric video, custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license, with one task instruction per episode. Generated or synthetic video, point tracks, optical flow, 2D or 3D trajectory labels, keypoint labels, robot trajectories, teleoperation data, synthetic data and language annotations beyond that instruction are not Firsthand deliverables.

Per the paper, the key insight is to cast motion prediction from web data as generic zero-shot video prediction. Given an image of the scene and a language description of the task, a pre-trained video generation model produces a video of a human doing the task. A single closed-loop robot policy is then conditioned on that generated video. The authors generate human video rather than robot video because, in their words, current video models cannot generate robot videos zero-shot and need robot-specific fine-tuning to do so.

There are two stages. First, the video model generates a human video from the scene image and the task text. The authors use VideoPoet without any fine-tuning, with prompts of the form “A person task-name, static camera”. Second, a closed-loop policy takes the generated video and the last few robot observations and predicts end-effector actions, trained with a behavior cloning loss over discretized action bins.

During training only, the authors add an auxiliary track prediction loss. An off-the-shelf tracker extracts point tracks from the generated video and the robot video, and a transformer must predict them from the policy’s latent tokens. The paper states this track prediction is not used at test time, which reduces test-time computation.

Per the paper, the policy is trained on an existing offline dataset of robot demonstrations from earlier work (RT-1), augmented with some paired human-and-robot demonstrations from Vid2Robot. For each robot demonstration, the authors generate a matching human video offline from the first frame and the language instruction, which they call fully automatic. They also report a co-training experiment with roughly 400 additional teleoperated trajectories. These are the authors’ descriptions, and Firsthand has not verified them.

The human video in this pipeline is generated, not recorded. Real human capture does not appear as an input to the method as the paper describes it.

The authors evaluate four levels of generalization: mild, standard, object-type and motion-type. In their Table I, Gen2Act averages 60% success across the four levels, against 22% for RT-1, 37% for Vid2Robot and 49% for Gen2Act without the track loss; on motion-type generalization it reports 30%. The introduction states roughly 30% higher absolute success than the most competitive baseline on novel object and motion types. With co-training on additional teleoperated data they report a 64% average.

For chained long-horizon activities such as making coffee, they report success falling from stage to stage over five trials. These are the authors’ figures, measured on their robot and tasks; Firsthand has not verified or reproduced them.

Per the paper, the system is limited by current video generation models, for example their inability to generate realistic hands, which limits very dexterous tasks. At the higher generalization levels, when the generated video is implausible and robot data support is thin, the policy tends to fail, so failures correlate with generation errors. A plausible video does not guarantee success, since grasping and following the object path after a grasp can still go wrong. The authors suggest recovering denser motion information, such as object meshes, as future work.

The first table sets Gen2Act beside three site entries. The Gen2Act paper cites Track2Act and describes point-track predictors trained from web video as hard to scale because they depend on intermediate models for ground truth. It does not cite Im2Flow2Act, and it names RT-2 and OpenVLA only as examples of policies co-trained with vision-language models. Its comparisons are the authors’ own.

Per the paper, the method needs a pre-trained video model, a robot demonstration dataset with language instructions, and an off-the-shelf point tracker. The robot side is the data a buyer would have to supply, and the paper does not state whether real egocentric human capture would help, so that is Not stated.

Firsthand states real human egocentric capture with one task instruction per episode and depth derived from a calibrated stereo rig. That is a note about format, not a claim of compatibility. Generated video and any tracks would come from tools the buyer runs, not from Firsthand.

When a program needs real human demonstrations that public or synthetic data does not cover, specified field by field. Firsthand does not build or train models. Confirm the fields on a Spec Pilot, a small fast first batch; the median from signed spec to first validated batch is 48 hours where coverage is deep, three to four weeks where thin.

What is Gen2Act?

A published framework (arXiv 2409.16283) that generates a human video of a task with a pre-trained video model, then runs a robot policy conditioned on that generated video. It is third-party research.

Does Gen2Act fine-tune the video model?

No. Per the authors, they use a pre-trained VideoPoet model directly, without any fine-tuning, to generate human videos.

What results do the authors report?

An average of 60% success across four generalization levels in their Table I, against 22% for RT-1 and 37% for Vid2Robot. These are the authors’ figures; Firsthand has not verified them.

How does Gen2Act differ from Track2Act?

Track2Act predicts 2-D point tracks from an initial and a goal image. Per the Gen2Act authors, Gen2Act generates a human video and uses point tracks only as a training-time auxiliary loss.

Does Firsthand use Gen2Act, and can I request a custom dataset?

Firsthand is not affiliated with the Gen2Act authors and claims no use, training or compatibility. Generated or synthetic video, point tracks, trajectory labels, robot trajectories and teleoperation data are not Firsthand deliverables. You can request a custom dataset.