What Is LLaRA?

LLaRA is introduced in LLaRA: Supercharging Robot Learning Data for Vision-Language Policy (arXiv 2406.20095), first submitted on 28 June 2024 and last revised on 30 January 2025 by Xiang Li and colleagues. The arXiv listing notes ICLR 2025, and the paper lists Stony Brook University and the University of Wisconsin-Madison. The abstract says that directly adapting a pretrained vision-language model for robotic control remains challenging, particularly with a limited number of robot demonstrations. LLaRA is the authors’ answer: a framework that formulates the robot action policy as visuo-textual conversations, motivated by visual instruction tuning in computer vision. This page draws only on the paper, the GitHub repository and the Hugging Face paper page. No separate project page and no Hugging Face model card were read.

The paper starts from a behavior cloning dataset of expert trajectories. Each state-action pair becomes a single-round conversation, called Instruct-BC (inBC). The current image and the textual task description form the user instruction, and the action is written as text. Instead of the special tokens for quantized actions used by RT-2 and OpenVLA, LLaRA uses 2D image coordinates normalized to the range 0 to 1, with rotation written as a number of degrees. The query also carries the history of executed actions, and the response plans several future steps, although only the first is executed. A second variant, D-inBC, replaces reference images in the task description with object detection results written as text, because the base model takes one image per conversation. The six auxiliary datasets are localization, detection, action prediction, future prediction, spatial relationship and temporal relationship. The paper says they are generated from the same trajectories with scripted templates and no additional action annotations.

LLaRA starts from a pretrained LLaVA-1.5-7B model, whose design is a visual encoder, a multilayer perceptron adapter and a language model. Training is a single-stage finetune that follows the original LLaVA stage 2 settings and updates every parameter except the vision encoder. The paper reports a single-cycle cosine schedule, a 0.03 warm-up ratio and a maximum learning rate of 2e-5, with batch size 32 for the 0.8k and 8k subsets and 128 for 80k.

At inference a new conversation starts at each step, sampling is disabled, and the text output is parsed for coordinates and a rotation angle. A predefined mapping, estimated by visual calibration in simulation and in the real world, converts image coordinates into robot action space. The paper tests three object detectors: the policy model itself, a Mask R-CNN with a ResNet50 backbone, and ground-truth oracle boxes. The real-world table caption names OWLv2-base-16 as the detector.

The simulated source is VIMA-Bench, a table-top manipulation environment with 17 tasks. Its full dataset has 660k expert trajectories covering 13 of the 17 tasks, and the paper subsamples it into VIMA-0.8k, VIMA-8k and VIMA-80k. Each episode has two third-person cameras, but LLaRA keeps only the front view. The Figure 1 caption describes the real-world demonstration as built from eight thousand simulated trajectories, six auxiliary datasets and about one thousand additional in-domain images.

The real-world setup is an xArm7 arm with a gripper and a Logitech C140 webcam above it, giving a third-person view. The action space is 4 DoF for pick-and-place positions plus 1 DoF for rotation. The authors build two in-domain datasets, xArm-Det and xArm-Action, each of 2,447 single-turn conversations.

In simulation, models are tested on VIMA-Bench levels L1 to L3, with 20 random seeds per task. The authors do not report L4, because the training set recorded zero rotation for all sweeping tasks. The paper says its instruction-style data consistently beats the RT-2 Style baseline, and that RT-2 Style at 100% of the data still trails its own method trained on 12%. Its Table 13 lists the best configuration at 12% of the data above VIMA trained on all 660k trajectories.

Real-world tests use three protocols: zero-shot, finetuning and joint training. With minimal finetuning on xArm-Action, the paper reports 75% success on its distractor task, against 10% zero-shot. It also states limits. The auxiliary data helps most when training data is limited and sometimes adds little at VIMA-80k. The authors suspect it may even distract the model there. Feeding several images into one conversation hurt results in their ablation.

The GitHub repository LostXine/LLaRA is the official implementation, and it lists an Apache License 2.0. The README gives the same Python environment as LLaVA, a revised LLaVA package and VIMABench setup. It states minimum hardware: one GPU with 24GB for inference, one 24GB GPU for LoRA finetuning, and at least 300GB of system RAM plus four Ampere GPUs with 24GB each for full-parameter finetuning. It points to a pretrained checkpoint on Hugging Face, but that model card was not read. Licence terms for the paper, datasets and checkpoints beyond the repository label are Not stated in the sources read. This is not legal advice.

LLaRA is a data conversion recipe plus a training setup. Its inputs are robot behavior cloning trajectories, simulated VIMA data and xArm-specific datasets, all with robot actions. Firsthand supplies a different thing: real human egocentric demonstrations from a head-mounted camera and two wrist cameras. Nothing on this site describes robot action or joint data, end-effector pose labels, teleoperation data, simulation data, visual question answering data, instruction-tuning conversations or video captions, and none is claimed here. The only language field the site states is one task instruction per episode.

Firsthand makes no claim that its data trained, was used with or is compatible with LLaRA, LLaVA, VIMA-Bench or any other model, robot, simulator or benchmark. Firsthand data is real capture, never synthetic, generated or simulated. The site’s general view is in its guide on training data for VLA models, linked below.

Firsthand collects real egocentric demonstrations on demand, to a written spec. The rig is a head-mounted global-shutter camera plus two wrist cameras, with active-stereo depth and 3x IMU on one hardware-triggered PTP clock. Episodes carry 3D hand pose at 21 joints per hand with a per-joint visibility flag, wrist trajectories, contact events, frame-accurate action segments and one task instruction per episode.

You write the spec, confirm it on a Spec Pilot, and pay per validated hour. The site states a 48-hour median turnaround where coverage is deep and three to four weeks where it is thin. Delivery is RLDS, LeRobot, WebDataset or HDF5, with a buyer-owned commercial license. Request a custom dataset below.

What is LLaRA?

Large Language and Robotics Assistant, a framework that formulates a robot action policy as visuo-textual conversations so a pretrained vision-language model can be finetuned into a vision-language-action policy. It is introduced in arXiv 2406.20095.

How does LLaRA represent robot actions?

As text. The paper writes pick and place positions as normalized 2D image coordinates and rotation as degrees, rather than using special tokens for quantized actions. A predefined mapping converts the coordinates into robot action space.

What are the six auxiliary datasets?

Localization, detection, action prediction, future prediction, spatial relationship and temporal relationship. The paper says they are generated from existing trajectories with scripted templates and no additional action annotations.

Is LLaRA built for small amounts of robot data?

The abstract frames the problem as adapting a pretrained model with a limited number of robot demonstrations. The paper reports its approach helps most when data is limited, and that the auxiliary data adds less at the 80k subset.

Can LLaRA be used commercially?

The GitHub repository lists an Apache License 2.0. Licence terms for the paper, datasets and checkpoints beyond that label are Not stated in the sources read. Check the repository directly; this is not legal advice.

Does Firsthand use LLaRA, and can I request a custom dataset?

Firsthand is not affiliated with LLaRA’s authors and makes no claim of use, training or compatibility. Firsthand data is real human egocentric capture. You can request a custom egocentric dataset: write a spec, confirm it on a Spec Pilot, pay per validated hour.