What Is LIV (Language-Image Representations and Rewards)?

The paper is by Yecheng Jason Ma, William Liang, Vaidehi Som, Vikash Kumar, Amy Zhang, Osbert Bastani and Dinesh Jayaraman, from the University of Pennsylvania and Meta AI; the project page and README name ICML 2023. The authors name three needs for a control-aware vision-language representation: align the two modalities, capture task progress grounded in language, and allow both broad pre-training and fine-tuning on small domain-specific datasets, which they say are scarce and expensive in robotics. LIV is their single objective for all three. The name expands as Language-Image Value learning in the abstract; the title says Representations and Rewards.

Firsthand is not affiliated with the LIV authors and makes no claim that any model, LIV included, was trained on Firsthand data. Firsthand captures real human egocentric video as custom collection, fast and on demand, to a written spec, with one task instruction per episode, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license. Captions, narrations, pretrained representations, embeddings, reward labels, value functions and robot data are not Firsthand deliverables, and language beyond that one instruction is not stated.

Per the paper, LIV extends VIP to goals given as language. It combines a VIP-style value term on images with a CLIP-style image-text contrastive term, and the authors show that the contrastive term is a special case of the video objective when a video is a repeated final frame. The value is implicit: similarity in the shared embedding space, using a cosine similarity scaled to be a valid value. The code follows the CLIP architecture with a ResNet50 vision encoder, initialised from CLIP weights. The equations and hyperparameters are Not stated here; see the paper.

Given a goal, the paper assigns a potential-based reward at each step: the change in similarity to the goal between consecutive frames. The abstract says the pre-trained model can assign dense rewards to each frame of videos of unseen robots or humans attempting a task in unseen environments, from only a language or image goal. The project page warns that zero-shot rewards do not always work on unseen robot domains and tasks because of language grounding and visual domain gaps, and shows fine-tuning as the remedy.

The authors pre-train on EPIC-KITCHENS-100, using only its RGB frames and text annotations, and describe it as a text-annotated egocentric video dataset of humans completing tasks in kitchens. They state it has 90k video segments, totalling 20M frames and 20k unique text annotations, and that no action labels are needed. This site’s EPIC-KITCHENS entry covers the dataset and names its video and annotation licences as non-commercial.

The abstract says that on several simulated and real-world robot environments, LIV models consistently outperform the best prior state representations for imitation learning and prior reward specification methods. The paper evaluates two simulated suites and one real Franka toy-kitchen setup, and shows the same objective fine-tuning LIV and other models, such as CLIP, on small in-domain datasets. These are the authors’ claims on their setup; Firsthand has not verified them, and no scores are repeated here.

The second table sets LIV beside three site entries, summarising what those entries state. The paper treats VIP as its base and R3M as a baseline, and cites Voltron’s authors as concurrent work.

Per the paper and README, pre-training needs action-free video with text annotations, and fine-tuning needs a small in-domain set of text-annotated videos, stored as numbered frames plus a manifest.csv. Firsthand states real human egocentric capture with one task instruction per episode. That is not a compatibility claim: any frame conversion or captions would be the buyer’s. For real human demonstrations that public video does not cover, request on-demand custom collection and confirm it on a Spec Pilot, with a median of 48 hours to a first validated batch where coverage is deep.

What is LIV (Language-Image Representations and Rewards)?

A published method (arXiv 2306.00958), called Language-Image Value learning in the abstract, that learns one vision-language representation and reward from action-free, text-annotated video. It is third-party research.

How does LIV give a vision-language reward for robotic control?

Per the paper, a dense reward comes from the change in embedding similarity to a language or image goal between consecutive frames. The project page notes zero-shot rewards do not always work on unseen robot domains.

Does LIV pretrain on egocentric video such as EPIC-KITCHENS?

Yes. The authors pre-train on EPIC-KITCHENS-100 RGB frames and text annotations. This site’s entry names that dataset’s licences as non-commercial.

Can I use the LIV code?

The official GitHub README states the source code is under the MIT License. Check the licences of any dataset you train on.

Was LIV trained on Firsthand data, and can I request a custom dataset?

No. Firsthand is not affiliated with the authors and claims no use or compatibility. The only language stated is one task instruction per episode. You can request a custom dataset.