What Is the CALVIN Benchmark?

CALVIN, short for Composing Actions from Language and Vision, is introduced in CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks (arXiv 2112.03227) by Oier Mees, Lukas Hermann, Erick Rosete-Beas and Wolfram Burgard of the University of Freiburg and the University of Technology Nuremberg. The project page lists it in IEEE Robotics and Automation Letters (RA-L), 2022, and the GitHub README says it won the 2022 RA-L Best Paper Award. The abstract argues that robots coexisting with humans must relate language to perception and action, and must acquire general-purpose skills that compose into long-horizon tasks through unconstrained instructions. The paper names three components: an environment, a dataset and a challenge.

CALVIN is third-party research. Firsthand is not affiliated with its authors, makes no claim that any named model was trained or evaluated on CALVIN using Firsthand data, and makes no claim that Firsthand data is compatible with or used by CALVIN. Firsthand captures first-person human video, collected fast, on demand and to a written spec. It is not robot or teleoperation data. The fields the site states are head and wrist camera video, depth derived from a calibrated stereo rig, 3D hand pose at 21 keypoints per hand with a per-joint visibility flag, wrist trajectories, action segments, and 6-DoF IMUs at 200 Hz on one clock. The only language field is one task instruction per episode.

The paper describes four structurally related environments, A, B, C and D. Each has a 7-DOF Franka Emika Panda arm with a parallel gripper, and a desk with a sliding door and a drawer that open and close, a button that toggles a green light, a switch that controls a light bulb and three rectangular blocks of different colours and shapes. The environments differ in texture, and the door, drawer, button and switch sit in different places. The desk, robot and static camera stay put. Physics run in PyBullet, and objects are reduced to unicoloured primitive shapes.

The paper defines 34 tasks that can be achieved in every environment, such as open drawer, lift red block, stack blocks, unstack blocks and turn on light bulb. The environment detects which task was completed from the change in state between the first and last step of a sequence, which can serve as a sparse reward. The project page’s leaderboard tables head the task column as 32 tasks, so the sources differ; the paper states 34.

The paper describes about 24 hours of teleoperated, unstructured play data collected with an HTC Vive VR headset, about six hours in each environment, or about 2.4M interaction steps. Three untrained people collected it with one guideline: explore the environment without dropping objects from the table. The paper calls play data free-form, with no task categories, and says it contains exploratory and sub-optimal behaviors such as retrying.

Language is sparse by design. The paper collects over 400 crowd-sourced instructions for the 34 tasks, labels episodes procedurally from the recorded environment state and annotates only 1% of the recorded interaction data, with precomputed MiniLM language embeddings included. Sensors are RGB-D from a static camera and a gripper camera, a vision-based tactile image and proprioceptive state. Actions can be absolute cartesian, relative cartesian or joint, sent at 30 Hz. The paper contrasts this suite with prior work that relies on RGB images from an egocentric camera.

The paper offers three train and test combinations of rising difficulty: a single environment, multi environment (train on all four, test in one) and zero-shot multi environment (train on three, test on the fourth, unseen one). Two metrics apply to all three. Multi-Task Language Control (MTLC) runs 10 rollouts for each of the 34 tasks from valid unseen starting states, with test instructions that are not in the training set.

Long-Horizon MTLC (LH-MTLC) chains five tasks in a row, using 1000 unique instruction chains filtered for cycles, redundancy and similarity. Only feasible sequences from a predefined initial state are allowed. The robot is reset to a neutral position after every sequence, so the policy must rely on language, and it moves to the next instruction only when the environment state confirms the current task. The authors report that their multi-context imitation learning (MCIL) baseline reaches up to 53.9% success on short-horizon tasks and performs poorly over long horizons. The project page keeps a leaderboard of other methods; those figures are not reproduced here.

Both are simulated benchmarks with provided human-teleoperated data and code under the MIT License, and they ask different questions. CALVIN’s sources centre on chaining unconstrained language instructions in a family of desk scenes with one robot arm. The site’s LIBERO entry states 130 tasks in four suites built to study lifelong knowledge transfer, with a sparse reward and a focus on lifelong imitation learning. The two use different tasks, metrics and suites, so scores are not comparable across them.

The sources read for this entry do not discuss vision-language-action models, so why papers on those models use either benchmark is Not stated in the sources read. The CALVIN README lists open-source models that outperform its baselines; this entry reproduces none of them or their scores.

CALVIN is simulated, with unicoloured primitive objects, and the authors say they would extend it to more realistic and diverse objects if future advances require new challenges. The paper says the benchmark captures many challenges of real-world settings and that its sensor suite should help researchers build agents for real-world tasks, but results on physical robots are Not stated in the sources read. The site’s sim-to-real gap entry says simulators approximate physics, lighting and material behavior, and that real capture matches deployment conditions because it is real.

A benchmark also fixes its scene and tasks: a desk, a drawer, a door, a light and three blocks. It cannot tell a program whether its own tasks, environments or people are covered by training data. That question belongs to a data spec, not a leaderboard.

CALVIN is a simulated benchmark with play data from its authors, not a source of human video. The MIT License is stated for its code, environments and baselines; a separate dataset licence is Not stated in the sources read. When a program needs first-person human video from particular tasks, environments or people, write them into a spec and confirm it on a Spec Pilot, a small fast first batch. Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks. Billing is per validated hour, and delivery is in the format your training stack already reads: RLDS, LeRobot, WebDataset or HDF5. Firsthand does not deliver robot or teleoperation data, so any robot demonstrations come from your own setup.

What is the CALVIN benchmark?

An open-source simulated benchmark for long-horizon, language-conditioned robot manipulation. A single agent follows a series of unconstrained language instructions in four tabletop environments, using 34 tasks and about 24 hours of teleoperated play data.

What do MTLC and LH-MTLC measure?

Multi-Task Language Control measures success on single tasks from the 34-task set, with 10 rollouts per task. Long-Horizon MTLC measures how many instructions in a row of five an agent completes across 1000 unique instruction chains.

How big is CALVIN?

The paper states four environments, 34 tasks, about 24 hours of teleoperated play data and about 2.4M interaction steps. Only 1% of the data carries language annotations. The project page leaderboard headings say 32 tasks, so the sources differ on that count.

Does CALVIN use real robots?

No. It is simulated in PyBullet with a Franka Emika Panda arm. Whether results transfer to physical robots is Not stated in the sources read.

Is CALVIN free to use?

The project page and GitHub repository state the MIT License for the code, environments and baselines. A separate dataset licence is Not stated in the sources read. Read the official pages yourself; this is not legal advice.

Does Firsthand use CALVIN, and can I request a custom dataset?

Firsthand is not affiliated with the CALVIN authors, makes no claim that any model was trained or evaluated on CALVIN using its data, and makes no claim that its data is compatible with or used by CALVIN. You can request a custom egocentric dataset: write the tasks, fields and format into a spec, confirm it on a Spec Pilot, and billing is per validated hour. The only language field is one task instruction per episode.