What Is the LIBERO Benchmark?

LIBERO is introduced in LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning (arXiv 2306.03310) by Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu and Peter Stone, with an official project page, GitHub repository, documentation and Hugging Face dataset repository. The abstract describes a benchmark of lifelong learning for robot manipulation that highlights five research topics: how to transfer declarative knowledge, procedural knowledge or a mixture of both; how to design effective policy architectures; how to design effective algorithms; how robust a lifelong learner is to task ordering; and the effect of model pretraining. The project page says the community lacked good testbeds for lifelong learning in decision making, and describes a benchmark that can continually grow. The full paper body was not read for this entry.

LIBERO is third-party research. Firsthand is not affiliated with its authors, makes no claim that any named model was trained or evaluated on LIBERO using Firsthand data, and makes no claim that Firsthand data is compatible with or used by LIBERO. Firsthand captures first-person human video, collected fast, on demand and to a written spec. It is not robot or teleoperation data. The fields the site states are head and wrist camera video, depth derived from a calibrated stereo rig, 3D hand pose at 21 keypoints per hand with a per-joint visibility flag, wrist trajectories, action segments, and 6-DoF IMUs at 200 Hz on one clock. The only language field is one task instruction per episode.

Per the abstract, lifelong learning aims at a generalist agent that learns and adapts over its lifespan. Image and text settings mostly involve transferring declarative knowledge of entities and concepts. Lifelong learning in decision making also needs procedural knowledge, such as actions and behaviors, to transfer. The README repeats the split as declarative knowledge about objects and spatial relationships and procedural knowledge about motion and behaviors.

The project page adds the motivation: robots may be pretrained with skills in a factory, yet should also learn with their human users throughout their lifetime. It names knowledge transfer as the core problem the benchmark invites researchers to study.

Per the README, LIBERO provides 130 tasks grouped into four suites: LIBERO-Spatial, LIBERO-Object, LIBERO-Goal and LIBERO-100. The first three have controlled distribution shifts, meaning each requires transfer of a specific type of knowledge. LIBERO-100 has 100 manipulation tasks that require transfer of entangled knowledge. It is split into LIBERO-90, for pretraining a policy, and LIBERO-10, for testing the agent’s downstream lifelong learning performance.

The abstract says tasks come from an extendible procedural generation pipeline that can in principle generate infinitely many, and the documentation lists scene creation and task generation sections. Task counts for the first three suites, and which type of knowledge each one targets, are Not stated in the sources read. The number of demonstrations per task is also Not stated; the sources say only that human-teleoperated demonstrations are provided for all tasks, stored as one HDF5 file per task on Hugging Face.

The README says the agent receives +1 when a task is finished, that only a sparse reward function is currently supported, and that because sparse-reward reinforcement learning is extremely hard, the benchmark mainly focuses on lifelong imitation learning. For benchmarking, each task suite fixes a set of initial states. Policies are evaluated on the fly during training, or with a separate evaluation script.

A run chooses a benchmark from LIBERO_SPATIAL, LIBERO_OBJECT, LIBERO_GOAL, LIBERO_90 and LIBERO_10, a policy from a BC RNN, BC Transformer or BC ViLT policy, and an algorithm from base, er, ewc, packnet and multitask. The abstract reports three findings: sequential finetuning outperforms existing lifelong learning methods in forward transfer, no single visual encoder architecture excels at all types of knowledge transfer, and naive supervised pretraining can hinder performance in later lifelong learning. How scores are aggregated, and any score values, are Not stated in the sources read, and none are reproduced here.

The sources read do not discuss vision-language-action models, so the authors’ view of why VLA papers use LIBERO is Not stated in the sources read. What they do state are practical properties: public code under the MIT License, datasets under CC BY 4.0, demonstrations for every task, fixed initial states, and a language instruction exposed for each task in the README’s example. A shared task set with provided demonstrations lets different papers train and evaluate on the same suites. That is an inference from those properties, not a claim by the authors.

Elsewhere on this site, the OpenVLA-OFT entry covers a paper that reports LIBERO results. Its figures belong to that entry and its sources, so none are reproduced here. Compare scores only against each paper’s own sources, because the sources read here do not say how other papers configure their runs.

The README’s scripts render tasks off-screen and set a MuJoCo rendering device variable, and its example sets camera height and width to 128, so LIBERO is a simulated benchmark. Whether results on it transfer to physical robots is Not stated in the sources read. The site’s sim-to-real gap entry says simulators approximate physics, lighting and material behavior, and that real capture matches deployment conditions because it is real.

A benchmark also fixes its tasks. It cannot tell a program whether its own tasks, environments or people are covered by training data. That question belongs to a data spec, not a leaderboard.

LIBERO is a benchmark with provided robot demonstrations, not a source of human video, and its code and datasets come from its authors. When a program needs first-person human video from particular tasks, environments or people, write them into a spec and confirm it on a Spec Pilot, a small fast first batch. Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks. Billing is per validated hour, and delivery is in the format your training stack already reads: RLDS, LeRobot, WebDataset or HDF5. Firsthand does not deliver robot or teleoperation data, so any robot demonstrations come from your own setup.

What is the LIBERO benchmark?

A benchmark of lifelong learning for robot manipulation from a June 2023 paper. It has 130 tasks in four task suites, a procedural generation pipeline, and human-teleoperated demonstrations for all tasks, and it is built to study knowledge transfer.

What are LIBERO-90 and LIBERO-10?

Per the README, LIBERO-100 has 100 manipulation tasks and is split into LIBERO-90 for pretraining a policy and LIBERO-10 for testing the agent’s downstream lifelong learning performance. Task counts beyond the names are Not stated in the sources read.

How many tasks and demonstrations does LIBERO have?

The README states 130 tasks in four suites. Demonstrations are provided for all tasks, but the number per task is Not stated in the sources read.

Is LIBERO free to use?

The README lists the codebase under the MIT License and the datasets under Creative Commons Attribution 4.0 International (CC BY 4.0). Read each licence yourself; this is not legal advice.

Does LIBERO use language instructions?

The README’s example retrieves a language instruction for each task. How instructions are written, and whether every suite uses them the same way, is Not stated in the sources read.

Does Firsthand use LIBERO, and can I request a custom dataset?

Firsthand is not affiliated with the LIBERO authors, makes no claim that any model was trained or evaluated on LIBERO using its data, and makes no claim that its data is compatible with or used by LIBERO. You can request a custom egocentric dataset: write the tasks, fields and format into a spec, confirm it on a Spec Pilot, and billing is per validated hour. The only language field is one task instruction per episode.