What Is Video Prediction Policy (VPP)?

VPP is introduced in Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations (arXiv 2412.14803) by Yucheng Hu, Yanjiang Guo, Pengchao Wang, Xiaoyu Chen, Yen-Jen Wang, Jianke Zhang, Koushil Sreenath, Chaochao Lu and Jianyu Chen. The abstract says “we propose the Video Prediction Policy (VPP)”, which names the method, and the arXiv comments list ICML 2025 Spotlight. Read: the arXiv abstract page, the full paper text, the official project page and the official GitHub README.

Firsthand is not affiliated with the VPP authors and makes no claim that any model, VPP included, was trained on Firsthand data. Firsthand captures human egocentric video, not robot data: custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license. Camera calibration files, multi-view rigs, 3D labels, reward labels, success or failure labels, rollouts, planning annotations and language annotations beyond one task instruction per episode are not Firsthand deliverables.

Per the paper, VPP learns in two stages. First, a general-purpose video foundation model is fine-tuned into a manipulation-focused text-guided video prediction (TVP) model, using internet human manipulation data, internet robot data and self-collected data.

Second, a Video Former aggregates features from the frozen TVP model into a fixed number of tokens, and a diffusion policy head generates the action sequence. The authors describe the result as an implicit inverse dynamics model conditioned on predicted future representations.

The authors give this name to the latent features inside a video diffusion model. They are a tensor that explicitly holds one current step and the predicted future steps, whereas earlier vision encoders, pre-trained on single images or image pairs, tend to capture static information.

For a robot with more than one camera view, such as a third-view and a wrist camera, the paper predicts the future for each view independently.

The paper uses the video diffusion model as a vision encoder rather than a denoiser, with a single forward step. Its reasoning is that denoising an entire video is time-consuming and can lead to open-loop control, while the first step already gives a rough trajectory of future states.

The authors report that each observation passes through the TVP model once, in under 160 ms, and that control reaches 7–10 Hz on a consumer RTX 4090 GPU with action chunking. These are their figures on their own setup.

The authors chose the open-source Stable Video Diffusion model, which they state has 1.5 billion parameters, added CLIP language features through cross-attention layers and set the output video to 16×256×256.

The GitHub README says VPP is developed from Stable-Video-Diffusion and the diffusion transformer policy MDT, and lists an MIT License. The official repository also holds the training and evaluation scripts.

These are the authors’ results on their own benchmarks and robots; Firsthand does not reproduce them and ranks no models. Their Table 1 reports an average task completion length of 4.33 on the CALVIN ABC→D benchmark, listed beside 3.06 for GR-1.

The abstract also states a 31.6% increase in success rates for complex real-world dexterous manipulation tasks. The arXiv abstract and the project page quote different relative CALVIN improvements, so this page repeats only the table value.

The comparison table below sets VPP beside four site entries. In short, VPP freezes a fine-tuned video prediction model and learns actions from its internal features, while the paper describes GR-1 as producing one image per forward pass without a pre-trained video foundation model.

The VPP paper also cites an earlier video-prediction model named Seer (arXiv 2303.14897). That is a different work from the site’s Seer entry, which covers arXiv 2412.15109.

The paper names internet human manipulation video, internet robot data and self-collected robot data for the video model, and says its real-world demonstrations were gathered by teleoperating robot arms. Whether any of that video is egocentric is Not stated.

Firsthand states human egocentric capture with stereo-derived depth, 21-joint hand pose and one task instruction per episode. Whether those streams match VPP’s inputs is Not stated, and no Firsthand data is claimed to have trained it.

When a program needs real human egocentric demonstrations that public data does not cover. Firsthand does not build or train models. Confirm the fields on a Spec Pilot, a small fast first batch; the median from signed spec to first validated batch is 48 hours where coverage is deep, three to four weeks where thin.

What is Video Prediction Policy (VPP)?

A published generalist robot policy (arXiv 2412.14803) that learns an implicit inverse dynamics model conditioned on predictive visual representations from a video diffusion model. It is third-party research.

What are VPP predictive visual representations?

Latent features inside a video diffusion model that hold the current step and predicted future steps. The authors read them with a single forward step instead of denoising a full video.

Which video diffusion model does VPP use?

The authors fine-tune the open-source Stable Video Diffusion model, which they state has 1.5 billion parameters, into a text-guided video prediction model for manipulation.

Is the VPP code available?

Yes. The project page links an official PyTorch implementation on GitHub, which lists an MIT License.

Is VPP a world model?

The sources read do not call it one. Its video model predicts future frames from an image and an instruction, and the site’s world model entry describes a model that predicts how a scene changes in response to an action.

Does Firsthand use VPP, and can I request a custom dataset?

Firsthand is not affiliated with the VPP authors and claims no use, training or compatibility. Camera calibration files, multi-view rigs, rollouts and success or failure labels are not Firsthand deliverables. You can request a custom dataset.