What Is OTTER?

OTTER is introduced in OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature Extraction (arXiv 2503.03734) by Huang Huang, Fangchen Liu, Letian Fu, Tingfan Wu, Mustafa Mukadam, Jitendra Malik, Ken Goldberg and Pieter Abbeel. The official GitHub README lists UC Berkeley and Meta as affiliations and ICML 2025 as the venue. The authors argue that fine-tuning a pre-trained vision-language model degrades its semantic alignment, because visual and language features reach the policy independently. OTTER instead keeps the vision and language encoders frozen and passes only the visual features that match the instruction.

OTTER is third-party research. Firsthand is not part of it or affiliated with its authors, and makes no claim that any model was trained on Firsthand data. What the site states is custom egocentric collection of real human demonstrations, fast and on demand, to a written spec: 3D hand pose at 21 joints per hand, calibrated stereo depth and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset with a buyer-owned commercial license. The only language data the site states is one task instruction per episode, which is the kind of text instruction OTTER takes as input. Mapping any delivery to OTTER’s action format is Not stated.

Per the abstract, existing approaches fine-tune pre-trained vision-language models because visual and language features are fed to the downstream policy independently, which degrades the pre-trained semantic alignments. The authors add that robot datasets are far less semantically diverse than the datasets those models were trained on, so fine-tuning can hurt performance on unseen objects and environments.

OTTER takes per-token text features from CLIP’s language encoder and patch features from the attention output of CLIP’s last vision attention layer. It computes their similarity, applies a softmax over the image patches, and uses the result to weight the patch features, giving one visual token per text token. Only the softmax temperature is learnable. The whole CLIP model stays frozen during training.

The authors describe this as separating task planning, which selects the relevant visual features, from action planning, which predicts the motion.

The text-aware visual features are pooled into one token per camera by learnable cross-attention with four queries, the text features are pooled into one token, and the robot’s proprioceptive state goes through a feed-forward network. The three are concatenated and fed to a causal transformer with 4 layers, 8 heads and a hidden dimension of 512, trained with a context length of 12 steps.

The model predicts the next 12 actions as delta end-effector poses. At execution the authors use temporal ensembling and an action horizon of 8 steps.

On a real Franka robot, in single-primitive pick-and-place, the authors report 68% success on training tasks and 62% on unseen tasks for OTTER, against 30% and 9% for fine-tuned OpenVLA and 15% and 12% for fine-tuned Octo. Across four primitives (pouring, drawer, poking, pick-and-place) on 150 unseen-task trials, they report a mean of 67% for OTTER and 0.6% for fine-tuned OpenVLA. On their LIBERO unseen tasks they report 61% for OTTER, 48% for OpenVLA and 26% for Octo.

These are the authors’ results on their own setups and baselines, and Firsthand does not reproduce them or rank models.

The site’s OpenVLA entry covers a 7B model that fine-tunes a Prismatic-7B vision-language model on 970k Open X-Embodiment trajectories, and its RT-2 entry covers a model that writes robot actions as text tokens. The OTTER authors call this direct feature passing and use fine-tuned OpenVLA as a baseline. OTTER keeps CLIP frozen and passes selected features to a separate policy transformer.

The authors collect human tele-operated robot demonstrations on a Franka: 724 pick-and-place demonstrations, and 1,185 demonstrations across the four primitives. They also use LIBERO simulation tasks with 50 demonstrations each. Variants pre-trained on Open X-Embodiment are reported as OTTER-OXE.

The authors also build 15.6k human hand motion trajectories, with RGB frames, language instructions and hand actions, from two existing human-hand datasets. They report that pre-training on them raised unseen-task success from 62% to 70% while training-task success went from 68% to 67%, and note the human data has different camera angles and no wrist view.

On the site, the only language data is one task instruction per episode. Stereo depth, IMU data and egocentric capture are not described in the sources read as part of OTTER’s training.

The authors name two limits: scaling to morphologies that SE(3) transforms cannot easily parameterise, such as multi-finger robot hands, and long-horizon tasks and more complex scenes, which they have not extensively explored. The GitHub repository is released under the Apache License 2.0, and its README says the paper’s experiments used an earlier Jax implementation. The README says the dataset is hosted on Hugging Face.

When a program needs real human egocentric demonstrations of tasks, environments or objects that public data does not cover, specified in writing. Firsthand does not build or train models.

Confirm the spec on a Spec Pilot, a small fast first batch. Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks. Billing is per validated hour.

What is OTTER?

A published vision-language-action model (arXiv 2503.03734) that keeps CLIP frozen and passes text-aware visual features to a policy transformer. It is third-party research.

How does text-aware visual feature extraction work?

Per the paper, CLIP text-token features are compared with image-patch features, a softmax over patches weights them, and the result gives one visual token per text token.

Does OTTER fine-tune CLIP?

No. The authors keep the whole CLIP model frozen; only the softmax temperature is learnable. They report that fine-tuning CLIP lowered unseen-task success in their ablation.

Is OTTER better than OpenVLA?

The authors report OTTER ahead of fine-tuned OpenVLA on their unseen-task evaluations. These are the authors’ results on their own setups, and Firsthand ranks no models.

Is OTTER open source?

The official GitHub repository, Max-Fu/otter, is released under the Apache License 2.0, and its README says the dataset is hosted on Hugging Face.

Does Firsthand data train OTTER?

No claim is made. Firsthand is not affiliated with its authors, and the site does not state that any model was trained on Firsthand data. The only language data the site states is one task instruction per episode.