What Is VIMA?

VIMA is introduced in VIMA: General Robot Manipulation with Multimodal Prompts (arXiv 2210.03094), an ICML 2023 paper by ten authors, with affiliations on the project page that include Stanford, NVIDIA, Caltech, Tsinghua and UT Austin. The authors observe that task specification in robotics comes in many forms, such as imitating one-shot demonstrations, following language instructions and reaching visual goals, and that these are often treated as separate tasks handled by specialized models. Their claim is that a wide spectrum of manipulation tasks can instead be written as multimodal prompts, giving one uniform sequence input and output interface. This page draws only on the paper, the project page, two official GitHub repositories and the Hugging Face model and dataset cards. VIMA is a simulation-trained research agent. The model card warns that its checkpoints may not directly generalize to other simulators or the real world, and real-robot results are Not stated in the sources read.

Per the paper, a multimodal prompt is an ordered sequence of arbitrarily interleaved text and images, where each element is either text or an image. It spans six task categories: simple object manipulation, visual goal reaching, novel concept grounding, one-shot video imitation, visual constraint satisfaction and visual reasoning. The categories are not mutually exclusive. A rearrangement task asks the agent to match a scene image, few-shot imitation embeds video frames in the prompt, and novel concept grounding introduces unfamiliar words such as “dax” and “blicket” and explains them with in-prompt images. Inside the model, prompts become sequences of text tokens and object tokens rather than one flat string.

Per the paper, VIMA learns a policy that chooses each action from the prompt and the history of past observations and actions. A frozen pretrained T5 encoder processes the prompt. Text goes through the T5 tokenizer and embeddings. Images of full scenes are parsed into objects by a domain fine-tuned Mask R-CNN detector, and each object, a bounding box plus a cropped image, is encoded by a bounding box encoder and a ViT into object tokens, with small MLPs bridging the non-text tokens to T5. Single-object images use a dummy bounding box.

A causal transformer decoder alternates cross-attention layers, which read the encoded prompt, with self-attention layers over the interaction history. Predicted action tokens are mapped to discretized poses of the robot arm. Training is behavioral cloning: minimize the negative log-likelihood of expert actions on an offline dataset, with no simulator access during training, plus object augmentation that injects false-positive detections. Seven models from 2M to 200M parameters were trained, with the T5-Base encoder (111M) held constant and excluded from that count. The paper adapts three baseline variants, VIMA-Gato, VIMA-Flamingo and VIMA-GPT, because no prior method worked out of the box with multimodal prompts.

VIMA-Bench is a simulation benchmark built by extending the Ravens robot simulator. It provides 17 representative tasks with multimodal prompt templates, procedurally instantiated into thousands of instances by combining textures and tabletop objects. Observations include RGB images from a frontal and a top-down view, ground-truth segmentation and bounding boxes, and an end-effector type indicator. The action space covers primitive skills such as pick and place and wipe, parameterized by end-effector poses; the benchmark README lists a pick pose and a place pose, each a 2D position plus a quaternion rotation. Scripted oracles that use privileged simulator state generate the expert demonstrations.

Evaluation uses a four-level protocol, each level strictly harder than the last: placement generalization, combinatorial generalization, novel object generalization and novel task generalization. Each task has a binary success criterion and the metric is average success rate. The abstract reports that VIMA beats alternative designs in the hardest zero-shot setting by up to 2.9 times the task success rate at equal training data, and by 2.7 times the best competing variant with 10 times less data. Those are the authors’ simulation results, and this page reproduces no other scores. Dataset size is stated differently across sources: 600K+ in the abstract, 650K in the paper body and README, and 600K for training plus 50K held out in the model card. The dataset card says 650K trajectories for 13 tasks, and why that differs from 17 is Not stated in the sources read.

VIMA and RT-1 both map observations plus a task specification to robot actions, but they differ in the specification, the visual input and the data. Per the RT-1 entry on this site, RT-1 takes a short sequence of images and a natural-language task description and was trained on over 130k real-world robot episodes. VIMA’s prompt can interleave text with images or video frames, its visual input is a set of detected objects rather than whole-image features, and its demonstrations are scripted simulation trajectories. The first comparison table sets the two side by side with the language-conditioned policy entry, using only each source or site entry.

A language-conditioned policy, as the site defines it, is one whose action output depends on a natural-language instruction as well as what the robot observes. VIMA’s prompts can contain text, so that definition covers part of what VIMA takes in. Its prompts can also contain images and video frames, and the language-conditioned policy entry notes that Octo can use goal images as an alternative to language. Whether the VIMA sources call it a language-conditioned policy is Not stated in the sources read, and this page does not either.

VIMA’s data is simulated and released by its authors; Firsthand is not affiliated with them and makes no claim that its data was used with, trained or is compatible with VIMA or VIMA-Bench. What the site states instead is custom egocentric collection of real human demonstrations to a written spec, fast and on demand: head and wrist camera video, calibrated stereo depth, 200 Hz IMU, 3D hand pose at 21 joints per hand with a per-joint visibility flag, wrist trajectories, action segments and contact events. The only language field is one task instruction per episode.

This page makes no claim that Firsthand provides simulation data, image-plus-text prompts or prompt annotations, and the site states no mapping from its fields to VIMA’s action space or token format. Confirm a spec on a Spec Pilot, a small fast first batch. The median from signed spec to first validated batch is 48 hours where coverage is deep and three to four weeks where it is thin. Billing is per validated hour, and delivery is RLDS, LeRobot, WebDataset or HDF5.

What is VIMA?

A transformer-based robot agent that follows multimodal prompts, which interleave text and images, and outputs robot arm actions autoregressively. It was introduced with the simulated VIMA-Bench.

What are VIMA multimodal prompts?

Ordered sequences of interleaved text and images, including video frames, that specify a task, such as rearranging objects to match a scene image or following a demonstrated motion.

What is VIMA-Bench?

A simulation benchmark on the Ravens simulator with 17 task templates that become thousands of instances, plus a four-level generalization protocol and an oracle-generated imitation dataset.

How large are the VIMA models and dataset?

Seven checkpoints from 2M to 200M parameters, with a T5-Base encoder excluded from that count. The sources give 650K trajectories, or 600K for training and 50K held out; the dataset card says 13 tasks.

Is VIMA open source?

Both official repositories are under the MIT licence and the dataset is CC BY 4.0. The checkpoint licence is Not stated in the sources read. Read the licence text; this is not legal advice.

Does Firsthand use VIMA, and can I request a custom dataset?

Firsthand is not affiliated with the VIMA authors and makes no claim of use, training or compatibility. You can request a custom egocentric dataset: write a spec, confirm it on a Spec Pilot, pay per validated hour.