What Is RoboFlamingo?

RoboFlamingo is introduced in arXiv 2311.01378, “Vision-Language Foundation Models as Effective Robot Imitators”, by Xinghang Li, Minghuan Liu and co-authors at ByteDance Research and three universities, with an official project page and GitHub repository. Read: abstract, full text, project page and README.

RoboFlamingo is third-party research. Firsthand is not affiliated with its authors and claims no use, training or compatibility. Firsthand captures first-person human video, not robot data, fast, on demand and to a written spec: 21 keypoints per hand with a visibility flag, wrist trajectories, action segments and one task instruction per episode.

It splits the work. OpenFlamingo reads the images and the instruction at every step: a vision transformer and perceiver resampler encode two camera views, a third-person camera and a gripper camera, and a decoder fuses them with the instruction. An explicit policy head carries history, an LSTM by default, and predicts a 7-DoF end-effector pose and gripper status.

Training is imitation learning on language-conditioned demonstrations: mean squared error on relative pose, binary cross-entropy on gripper state. The paper says it can be trained or evaluated on a single GPU server, and reports experiments on one 8-GPU A100 server.

The paper evaluates only on CALVIN, a simulation with a Franka Emika Panda arm: 34 tasks, sequences of up to five instructions. Trained on all four splits and tested on D, it reports an average completed length of 4.09 against 3.06 for HULC. Trained on A, B and C and tested on D, it reports 2.48 against 0.67 for HULC and 0.90 for RT-1. These are the authors’ own figures.

A vision-language model outputs no actions on its own; the policy head supplies them. The paper contrasts RoboFlamingo with RT-2, which it describes as using private models and co-fine-tuning on large vision-language data, and with SayCan and PaLM-E, which need separate low-level skill policies. RoboVLMs is a separate study the sources read do not discuss.

RoboFlamingo’s CALVIN data is simulated teleoperation with robot actions, which Firsthand does not deliver; egocentric human video as RoboFlamingo training data is Not stated in the sources read. Firsthand captures human first-person video with one task instruction per episode and action segments; no compatibility is claimed. Put tasks and signals in a written spec and confirm it on a Spec Pilot. Median from signed spec to first validated batch: 48 hours where coverage is deep, three to four weeks where thin. Formats: RLDS, LeRobot v2.1, WebDataset, HDF5.

What is RoboFlamingo?

A vision-language manipulation framework from arXiv 2311.01378, built on OpenFlamingo, with a policy head that predicts robot actions.

How does it turn a vision-language model into a robot policy?

OpenFlamingo reads images and the instruction each step; an LSTM policy head carries history and predicts pose and gripper status.

How is RoboFlamingo trained?

By imitation learning on language-conditioned CALVIN demonstrations, training only the resampler, cross-attention and policy head.

Is RoboFlamingo open source?

The official GitHub repository lists an MIT License, and its README says pre-trained models are on Hugging Face.

Was it tested on real robots or trained on egocentric human video?

The paper states it does not deploy on real-world robots. Egocentric human video as training data is Not stated in the sources read.

Does Firsthand use RoboFlamingo, and can I request a custom dataset?

Firsthand is not affiliated with the RoboFlamingo authors and claims no use, training or compatibility. You can request a custom dataset, confirmed first on a Spec Pilot.