What Is LoRA Fine-Tuning for VLA Models?

LoRA, short for Low-Rank Adaptation, is introduced in the paper LoRA: Low-Rank Adaptation of Large Language Models (arXiv 2106.09685). The authors note that as models grow, full fine-tuning, which retrains all model parameters, becomes less feasible, and that deploying independent fully fine-tuned copies of a 175B-parameter model is prohibitively expensive. Their answer is to freeze the pretrained weights and inject trainable rank decomposition matrices into each layer of the Transformer architecture. The OpenVLA paper (arXiv 2406.09246) applies the idea to a vision-language-action (VLA) model. Its abstract says prior work had not explored methods for efficiently fine-tuning VLAs for new tasks, and that OpenVLA can be fine-tuned on consumer GPUs via modern low-rank adaptation methods and served efficiently via quantization without a hit to downstream success rate.

LoRA is a training method from third-party research. Firsthand does not train, fine-tune or host models, and it makes no claim that its data was used to LoRA fine-tune OpenVLA or any other named model, and no affiliation with their authors. What Firsthand states is custom egocentric data collection, on demand and to a written spec. Each episode carries one task instruction, and the fields the site states include 21-joint 3D hand pose with a per-joint visibility flag, wrist trajectories, action segments, contact events, 200 Hz IMU and calibrated stereo depth. It is first-person human video, not robot or teleoperation data. How a team turns that into inputs for its own LoRA run is its own pipeline, and the sources read do not describe it.

Instead of updating every weight in a layer, LoRA keeps the pretrained weights frozen and trains a pair of small matrices whose product forms the update. The LoRA paper’s abstract describes these as trainable rank decomposition matrices injected into each layer of the Transformer, which greatly reduces the number of trainable parameters for downstream tasks.

For GPT-3 175B, the abstract reports that LoRA can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times compared with fine-tuning with Adam. It also reports model quality on par with or better than fine-tuning on RoBERTa, DeBERTa, GPT-2 and GPT-3, a higher training throughput and, unlike adapters, no additional inference latency. Those results are for language models. The abstract does not describe vision-language-action models.

The OpenVLA paper compares full fine-tuning with partial approaches and with LoRA applied to all linear layers of the model. It reports that LoRA gave the best trade-off between performance and training memory, matching full fine-tuning on the select Franka-Tabletop tasks it tested while fine-tuning only 1.4% of the parameters. It also reports that the LoRA rank had a negligible effect on policy performance and recommends a default rank of r=32. With LoRA, the paper says OpenVLA can be fine-tuned on a new task within 10 to 15 hours on a single A100 GPU, an 8x reduction in compute compared with full fine-tuning.

The official repository supplies the recipe. Its LoRA section uses vla-scripts/finetune.py through the Hugging Face transformers library, and the project’s own description says its fine-tuning script supports (quantized) LoRA via Hugging Face’s PEFT library. The README example fine-tunes the openvla-7b checkpoint with a LoRA rank of 32 on a single A100 with 80 GB of memory, and says a smaller GPU with at least about 27 GB of memory can be used by changing the batch size.

The README recommends LoRA if you do not have enough compute to fully fine-tune a 7B-parameter model. It describes full fine-tuning, which updates all 7.5 billion parameters through PyTorch Fully Sharded Data Parallel, as more involved. It recommends it only if you have sufficient compute, for example a full node of 8 A100 GPUs, and if LoRA is insufficient for your use case, for example when the fine-tuning distribution varies drastically from the pretraining distribution.

The README also carries a March 2025 update recommending that readers try a newer recipe called OFT, which the project describes as an optimized fine-tuning recipe. This page does not cover OFT, and the sources read here do not tie it to LoRA data requirements.

The README says a custom dataset outside the Open X-Embodiment mixture can be converted to the RLDS format, which its fine-tuning script supports, or loaded through a custom PyTorch Dataset wrapper, and that it recommends the RLDS route for most users. After conversion, the dataset is registered with the data loader through a dataset config and a dataset transform function. The same RLDS route is described for full fine-tuning.

In other words, the repository describes LoRA as a change in which weights are trained. It does not describe a different kind of dataset for it. Firsthand lists RLDS among its delivery formats. Whether Firsthand episodes fit a particular model’s registered action space is Not stated, and Firsthand makes no claim about it.

Write the target skills, environments and signals into a skill spec and confirm them on a Spec Pilot, a small fast first batch. This is on-demand custom collection, not a catalog download.

Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks. Billing is per validated hour.

What is LoRA fine-tuning for VLA models?

A way to adapt a pretrained vision-language-action model by freezing its weights and training small low-rank matrices, so far fewer parameters are updated than in full fine-tuning.

Can you fine-tune OpenVLA with LoRA?

Yes. The OpenVLA paper reports that it can be fine-tuned on consumer GPUs via low-rank adaptation, and the official repository includes a LoRA fine-tuning script and example.

Does LoRA match full fine-tuning for VLAs?

In the OpenVLA paper, LoRA matched full fine-tuning on the select Franka-Tabletop tasks tested while updating 1.4% of the parameters. That is the authors’ result on their own tasks.

What LoRA rank should you use for OpenVLA?

The paper found rank had a negligible effect on policy performance and recommends a default of r=32. The README example also uses 32.

Does LoRA need a different kind of dataset?

The repository describes the same RLDS dataset route for LoRA and full fine-tuning, so LoRA changes which weights are trained rather than the dataset format.

Is Firsthand data used to LoRA fine-tune any model?

Firsthand makes no claim that its data was used to fine-tune OpenVLA or any other named model, and it is not affiliated with their authors. It collects custom egocentric data to a spec.