What Is LLARVA?

LLARVA is introduced in LLARVA: Vision-Action Instruction Tuning Enhances Robot Learning (arXiv 2406.11815) by Dantong Niu, Yuvan Sharma, Giscard Biamby, Jerome Quenum, Yutong Bai, Baifeng Shi, Trevor Darrell and Roei Herzig, first submitted on 17 June 2024. The project page lists Berkeley AI Research, UC Berkeley, and CoRL 2024. The abstract starts from a gap: instruction-tuned large multimodal models have succeeded at tasks such as image captioning and visual question answering, yet leveraging them for robotics remains an open question, and prior models trained on language and action data often generalized less well than desired. LLARVA is the authors’ response, a model trained with a novel instruction tuning method built on structured prompts.

Firsthand is not affiliated with the LLARVA authors and makes no claim that any model, LLARVA included, was trained on Firsthand data. Firsthand captures human egocentric video, not robot data: custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license. Visual traces, 2D trajectory labels, instruction-tuning data, keypoint labels, robot trajectories, teleoperation data, synthetic data and language annotations beyond one task instruction per episode are not Firsthand deliverables.

The abstract frames the problem as how to use instruction-tuned large multimodal models (LMMs) for robot learning. Its answer has two parts: a vision-action instruction tuning method that uses structured prompts to unify a range of robotic learning tasks, scenarios and environments, and an intermediate 2-D representation that the authors call a “visual trace”. The sources read do not spell out the letters of the acronym, so this entry does not expand it.

In the glossary sense, this places LLARVA beside other vision-language-action work: an image and a language instruction go in, and an action comes out. What sets it apart in the abstract is that the instruction is structured and that the model also writes down where the end-effector is going to travel in the image.

Per the project page, the model works from a language instruction that contains the robot model, the control mode, the robot task, proprioceptive information and the number of predicted steps. It outputs text with the next robot action or actions, together with the visual trace for the remainder of the episode.

Putting those fields into the prompt is what lets one model be tuned across robots and control modes rather than per setup. The abstract states that the structured prompts unify a range of robotic learning tasks, scenarios and environments. Which fields matter most is Not stated in the sources read.

The abstract says that predicting intermediate 2-D representations, which the authors refer to as “visual traces”, can help further align vision and action spaces for robot learning. The project page adds that the trace helps localize the end-effector. For each image in an episode, the authors compute the 2-D visual trace of the end-effector for the rest of the episode, using a bounding box detector trained on each of the different end-effectors.

The project page also describes two qualitative observations. The model can plan an alternative path that still succeeds, and the trace can act as a memory buffer for long-horizon tasks, compensating for the limited number of previous robot states the model can handle. These are the authors’ qualitative examples, not benchmark results.

The project page describes the pipeline. An input image goes through a frozen vision encoder, and an MLP layer projects the visual features into a latent space so they match the dimensionality of the language tokens. The language input is tokenized, the visual tokens and word tokens are concatenated, and both are fed to the auto-regressive transformers of the LMM, which are trained for next-token prediction.

The sources read do not give parameter counts, and this entry repeats none.

The abstract says the authors generated 8.5M image-visual trace pairs from the Open X-Embodiment dataset to pre-train the model, and evaluated it in the RLBench simulator and on a physical Franka Emika Panda 7-DoF robot. The project page describes the pre-training set as drawn from 37 Open X-Embodiment subsets covering 13 robots, and says three downstream tasks (“pick blue cube”, “stack the cubes” and “destack cube”) were used to fine-tune on a real robot. These figures are the authors’, and Firsthand has not verified them.

The arXiv abstract and the project page give different counts of RLBench tasks, so this entry repeats neither. It also reproduces no success rates. The Open X-Embodiment entry on this site describes that dataset as robot-collected data, not egocentric human video, which means LLARVA’s pre-training source differs in kind from Firsthand capture.

The comparison table below sets LLARVA beside two site entries and the general vision-language-action entry. The names are close but the works are separate: LLaRA is a different paper by different authors. The sources read for LLARVA do not discuss LLaRA or RT-Trajectory, so relationship cells say Not stated where that is the case. The table compares what each entry states and ranks no methods.

Per the sources read, LLARVA’s tuning data are images, structured instructions with robot and control fields, actions and 2-D end-effector traces, built from robot datasets. Whether human egocentric video would help a vision-action instruction-tuning pipeline is Not stated in the sources read.

Firsthand states human egocentric capture with stereo-derived depth, 21-joint hand pose and one task instruction per episode, and no visual traces, 2D trajectory labels, instruction-tuning data or robot trajectories. No Firsthand data is claimed to have trained or been used by LLARVA.

When a program needs real human egocentric demonstrations that public or synthetic data does not cover. Firsthand does not build or train models. Confirm the fields on a Spec Pilot, a small fast first batch; the median from signed spec to first validated batch is 48 hours where coverage is deep, three to four weeks where thin.

What is LLARVA?

A published method (arXiv 2406.11815, CoRL 2024) that instruction-tunes a large multimodal model on structured prompts for robot learning and predicts the next action plus a 2-D visual trace. It is third-party research.

What is a visual trace in LLARVA?

Per the abstract and project page, an intermediate 2-D representation of the end-effector path in the image, which the authors say helps align vision and action spaces.

What does the LLARVA prompt contain?

Per the project page, the robot model, control mode, robot task, proprioceptive information and number of predicted steps. The model answers with the next robot action(s) and the visual trace for the rest of the episode.

What data did the authors use to pre-train LLARVA?

Per the abstract, 8.5M image-visual trace pairs generated from the Open X-Embodiment dataset. These are the authors’ figures; Firsthand has not verified them and no Firsthand data is part of that set.

Does Firsthand use LLARVA, and can I request a custom dataset?

Firsthand is not affiliated with the LLARVA authors and claims no use, training or compatibility. Visual traces, 2D trajectory labels, instruction-tuning data, robot trajectories and rollouts are not Firsthand deliverables. You can request a custom dataset.