What Is VIP (Value-Implicit Pre-training)?

The paper is by Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar and Amy Zhang, with affiliations listed as FAIR, Meta AI and the University of Pennsylvania. arXiv records it as submitted on 30 September 2022 and revised on 7 March 2023, and notes ICLR 2023 (Notable-Top-25%, spotlight). The authors say reward and representation learning are two long-standing challenges for learning robot manipulation skills from sensory observations. Learning a general visual representation from large offline human videos has emerged as a promising path, but they say how those videos can be used for general-purpose reward learning remains open. VIP is their answer: one pre-trained representation that serves as both the visual encoder and the source of the reward.

Firsthand is not affiliated with the VIP authors and makes no claim that any model, VIP included, was trained on Firsthand data. Firsthand captures real human egocentric video, custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license, with one task instruction per episode. Pretrained representations, embeddings, reward functions, reward or value labels, robot trajectories, teleoperation data, paired human-robot video, synthetic data and language annotations beyond that instruction are not Firsthand deliverables.

Per the abstract, VIP casts representation learning from human videos as an offline goal-conditioned reinforcement learning problem. From that it derives a self-supervised dual goal-conditioned value-function objective that does not depend on actions, which is what allows pre-training on unlabeled human videos. The authors add that VIP can be understood theoretically as a novel implicit time contrastive objective that produces a temporally smooth embedding, so the value function is implicitly defined by embedding distance. The project page says it requires only an observation-only dataset, such as human videos, and that the core training loop can be as few as 10 lines of PyTorch. The loss details are Not stated here; see the paper.

The project page puts it simply: given the VIP embedding and a goal image, the reward is the embedding distance, or difference, to the goal. Because the representation is frozen and was not fine-tuned on in-domain or task-specific data, the abstract describes the reward as dense and smooth, usable for any goal-image-specified downstream task. The page calls this zero-shot reward specification on unseen robot tasks. The reward is computed from the authors’ pre-trained model; it is not data that a collection vendor supplies.

The abstract says VIP is trained on large-scale Ego4D human videos, a public egocentric dataset described on this site’s Ego4D entry and in its own official sources. The GitHub README states that its code supports training on Ego4D, using a config for the released model, and on any custom video dataset arranged as folders of numbered frame images. How much of Ego4D was used is Not stated in the sources read, so no Ego4D figures are repeated here.

The abstract says VIP’s frozen representation provides dense visual reward for an extensive set of simulated and real-robot tasks and significantly outperforms all prior pre-trained representations, and that it enables few-shot offline RL on real-world robot tasks with as few as 20 trajectories. The project page describes 36 vision-based manipulation tasks for trajectory optimization and online reinforcement learning, and four real-world tasks: CloseDrawer, PushBottle, PickPlaceMelon and FoldTowel. It shows side-by-side comparisons with R3M. These are the authors’ claims on their setup; Firsthand has not verified or reproduced them, and no scores are repeated here.

The second table sets VIP beside four site entries. The columns for the other entries summarise what those entries state, not new claims. The VIP sources read do not discuss VC-1 or MVP, so no head-to-head comparison with them is made. For the wider stage this belongs to, see the pretraining entry.

Per the sources read, pre-training needs only observation-only video, with no action labels, and the downstream step needs a goal image plus, for the offline RL results, a small set of robot trajectories. Firsthand states real human egocentric capture with stereo depth, 3D hand pose, IMUs and one task instruction per episode, delivered in RLDS, LeRobot format or WebDataset. The VIP repository expects frame folders, so any conversion would be the buyer’s work, and no compatibility is claimed. Training a representation or computing rewards is outside Firsthand’s scope.

When a program needs real human demonstrations that public video does not cover, specified field by field. Firsthand does not build or train models, or supply robot data. Confirm the fields on a Spec Pilot, a small fast first batch; the median from signed spec to first validated batch is 48 hours where coverage is deep, three to four weeks where thin.

What is VIP (Value-Implicit Pre-training)?

A published method (arXiv 2210.00030) that pre-trains a visual representation on Ego4D human video with a self-supervised, goal-conditioned value-function objective that does not depend on actions. The frozen embedding then gives a dense reward for a goal-image task. It is third-party research.

How does VIP produce a visual reward?

Per the project page, given the VIP embedding and a goal image, the reward is the embedding distance, or difference, to the goal. No fine-tuning on in-domain, task-specific data is used.

What results do the authors report?

The abstract reports that VIP significantly outperforms all prior pre-trained representations and enables few-shot offline RL on real-world robot tasks with as few as 20 trajectories. These are the authors’ claims; Firsthand has not verified them.

Can I use the VIP code?

The official GitHub README states the source code is licensed under CC BY-NC 4.0, which is a non-commercial license, and the repository page shows it as archived. Confirm the current terms yourself before relying on it.

Does Firsthand use VIP, and can I request a custom dataset?

Firsthand is not affiliated with the VIP authors and claims no use, training or compatibility. Pretrained representations, embeddings and reward functions are not Firsthand deliverables. You can request a custom dataset.