What Is VLA-RL?

VLA-RL is introduced in VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning (arXiv 2505.18719; submitted 24 May 2025, cs.RO) by Guanxing Lu, Wenkai Guo, Chubin Zhang, Yuheng Zhou, Haonan Jiang, Zifeng Gao, Yansong Tang and Ziwei Wang. It is a different paper from SimpleVLA-RL. Read: the arXiv abstract page, the GitHub README and the official blog the README links, of which the text retrieved ended partway through its implementation notes. The full paper text was not read.

Third-party research; Firsthand is not affiliated and claims no use, training or compatibility with it. Firsthand captures first-person human video, not robot data, fast, on demand and to a written spec: 21 keypoints per hand with a visibility flag, wrist trajectories and action segments, plus one task instruction per episode.

The abstract starts from the observation that high-capacity VLA models perform well on manipulation tasks by imitating human demonstrations, but that exploiting offline data with limited visited states causes execution failure in out-of-distribution scenarios. It proposes an exploration-based method that improves on online collected data at test time. VLA-RL is presented as an algorithmic and systematic framework that applies online reinforcement learning (RL) to pretrained auto-regressive VLAs in downstream tasks. Its first move is a trajectory-level RL formulation that models a manipulation trajectory as a multi-modal multi-turn conversation. The blog adds that the aim is to turn exploitation-based approaches into exploration-based ones.

To address sparse rewards, the abstract says the authors fine-tune a pretrained vision-language model as a robotic process reward model. It is trained on pseudo reward labels annotated on automatically extracted task segments. That is the source of the phrase VLA-RL robotic process reward. The README highlights describe the work as outcome-based multi-task reinforcement learning, and this page does not reconcile the two descriptions. Which vision-language model is used, how segments are extracted and how pseudo labels are produced are Not stated in the text read.

The abstract lists four implementation findings that improve stability and efficiency: a curriculum selection strategy, GPU-balanced vectorized environments, batch decoding and critic warmup. The blog describes the algorithm as Proximal Policy Optimization (PPO) with a rollout phase, advantage computation through Generalized Advantage Estimation and a clipped learning phase. It notes that the log-probability of an action is the sum of token-level log-probabilities in an autoregressive model, with 7 degrees of freedom in the OpenVLA action space. The README and blog name Ray, vLLM, LoRA and FSDP, and describe one GPU for inference with the other GPUs learning. The README calls the code a single-file implementation.

The abstract says VLA-RL enables OpenVLA-7B to surpass the strongest finetuned baseline by 4.5% on 40 challenging robotic manipulation tasks in LIBERO, and to match the performance of advanced commercial models such as π0-FAST. It also says VLA-RL benefits from increased test-time optimization, which the authors call an early spark of inference scaling laws in robotics. These are the authors’ claims, not Firsthand results, and this page repeats no other scores.

The site’s SimpleVLA-RL entry describes a separate paper, built on veRL, that fine-tunes an already supervised-fine-tuned VLA with simple binary outcome rewards. The OpenVLA entry describes a pretrained 7B model, which is the model VLA-RL improves, not an RL method. The post-training entry describes the final stage that narrows a model to one deployment task. VLA-RL is a method for improving a pretrained VLA with online RL; the sources read do not use the term post-training for it. The table ranks none of them.

The full paper was not read, so the reward model’s base model, the segment extraction and pseudo-labelling procedure, training hyperparameters and per-task results are Not stated here. The blog text retrieved ended partway through its implementation notes. The README describes the project as work in progress and lists real-world RL in the SERL style, more environments and more VLAs as to-do items. The arXiv author list has eight names and the README citation lists seven. Quote the paper for exact protocols.

Per the abstract, VLA-RL improves a pretrained VLA on data collected online, that is, from the policy’s own interaction, and a reward signal comes from the process reward model. Firsthand’s site states human video with depth derived from a calibrated stereo rig, 21 keypoints per hand with a visibility flag, wrist trajectories and action segments. Environments, reward models, rollouts and robot actions are Not stated as deliverables. The site’s only language data is one task instruction per episode.

The sources read describe RL on a pretrained robot policy, not human video. Firsthand records first-person human video to a written spec: depth from a calibrated stereo rig, 21 keypoints per hand with a visibility flag, wrist trajectories, action segments and one task instruction per episode. Whether such data suits the pretraining or supervised stage that precedes an RL stage is the buyer’s to check; the sources do not say. Confirm tasks and signals on a Spec Pilot. Median time to first validated batch: 48 hours where coverage is deep, three to four weeks where thin. Formats include RLDS, LeRobot format and WebDataset.

What is VLA-RL?

A published framework (arXiv 2505.18719) that uses online reinforcement learning to improve pretrained auto-regressive vision-language-action models, with a robotic process reward model for sparse rewards.

What is the robotic process reward model in VLA-RL?

Per the abstract, a pretrained vision-language model fine-tuned on pseudo reward labels annotated on automatically extracted task segments. Its base model and labelling procedure are Not stated in the text read.

How is VLA-RL different from SimpleVLA-RL?

They are different papers. The site entry describes SimpleVLA-RL as built on veRL with simple binary outcome rewards; VLA-RL uses a process reward model and applies PPO to OpenVLA-7B per the abstract and blog.

Does VLA-RL need new demonstrations?

The abstract describes improving a pretrained VLA on online collected data. Whether new human demonstrations are needed first is Not stated in the sources read.

Is VLA-RL open source?

The README lists the Apache-2.0 licence and describes a single-file implementation built on Ray, vLLM, LoRA and FSDP. It also describes the project as work in progress.

Does Firsthand use VLA-RL, and can I request a custom dataset?

Firsthand is not affiliated and claims no use, training or compatibility. The site states one task instruction per episode and no reward signals or robot actions. You can request a custom dataset, confirmed first on a Spec Pilot.