What Is RoboCat?

RoboCat is introduced in RoboCat: A Self-Improving Generalist Agent for Robotic Manipulation (arXiv 2306.11706) by Konstantinos Bousmalis, Giulia Vezzani, Dushyant Rao, Coline Devin, Alex X. Lee and colleagues, all at Google DeepMind, first submitted on 20 June 2023 and published in Transactions on Machine Learning Research in December 2023. The paper itself says the agent is “named RoboCat”, and the official DeepMind post explains that it is based on Gato, the Spanish word for cat. Read: the arXiv abstract, the full paper text and the official DeepMind blog post. Public weights and code are Not stated in them.

Firsthand is not affiliated with the RoboCat authors and makes no claim that any model, RoboCat included, was trained on Firsthand data. Firsthand captures human egocentric video, not robot data: custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license. Goal images, reward labels, success or failure labels, rollouts, planning annotations and language annotations beyond one task instruction per episode are not Firsthand deliverables.

Per the paper, RoboCat is a large transformer sequence model that builds on the Gato architecture. Images are tokenised with a pretrained VQ-GAN encoder, and proprioceptive observations and actions are tokenised following Gato. The authors describe it as a multi-task, multi-embodiment agent that handles different observation and action specifications natively, without a common action or observation representation.

The agent is trained with a token prediction loss.

RoboCat is visual goal-conditioned: a goal image shows an example of the task being solved, and the agent is evaluated only on task success. The authors note that any image in a trajectory can be relabelled as a hindsight goal for earlier steps, so goals can be extracted from existing data without extra human supervision.

This differs from language-conditioned models. The paper contrasts its approach with transformer policies conditioned on language, and the sources read describe no language prompt for RoboCat.

The DeepMind post lists five steps: collect 100 to 1000 demonstrations of a new task or robot using a human-controlled arm, fine-tune RoboCat on it to make a specialised spin-off agent, let that agent practise the task to generate more training data, add the demonstrations and self-generated data to the training dataset, and train a new version.

The paper says its experiments demonstrate one successful iteration of this process. The blog states that a spin-off agent practised a task an average of 10,000 times.

The authors describe three kinds of data: data from specialist reinforcement learning agents, mostly in simulation, human teleoperated expert data, mostly for physical tasks, and data generated by RoboCat itself. The paper counts 253 task variations, covering simulated and real Sawyer and Panda arms and a KUKA arm seen only during fine-tuning, with 36 real robots and 134 real objects in total.

The self-improvement pipeline also uses vision-based reward models, trained from binary annotations in which crowd annotators mark when a task is solved. All of this is the authors’ own robot data and tooling, not egocentric human video.

These are the authors’ results on their own setups; Firsthand does not reproduce them and ranks no models. The blog states that after 1000 human-controlled demonstrations on a new arm with a three-fingered gripper, RoboCat picked up gears successfully 86% of the time.

It also states that the initial version succeeded 36% of the time on previously unseen tasks after 500 demonstrations per task, and that the latest version more than doubled this on the same tasks.

The authors state their ultimate aim as a foundation agent for manipulation: a multi-embodiment agent trained on a large set of robot experience that adapts quickly through fine-tuning. The site’s robot foundation model entry describes the same pretrain-then-adapt pattern, and its cross-embodiment learning entry describes training on multiple embodiments.

The site’s Octo entry covers an open-source generalist policy with a diffusion action head, and its GR-1 entry a GPT-style model that learns from video prediction before robot fine-tuning. RoboCat differs from both: it is an autoregressive Gato-style agent with a self-improvement loop, and whether its weights are public is Not stated.

The paper considers vision-based tabletop object manipulation and only tasks whose success can be determined from the end state. Its robots work inside a standardised cage with a basket that defines the workspace. A fine-tuned specialist may lose performance on the original training tasks.

The sources read describe robot-arm data, teleoperated demonstrations and self-generated data, and do not mention egocentric human video, so that is Not stated.

Firsthand states human egocentric capture with stereo-derived depth, 21-joint hand pose and one task instruction per episode. Whether those streams match RoboCat’s inputs is Not stated, and no Firsthand data is claimed to have trained it.

When a program needs real human egocentric demonstrations that public data does not cover. Firsthand does not build or train models. Confirm the fields on a Spec Pilot, a small fast first batch; the median from signed spec to first validated batch is 48 hours where coverage is deep, three to four weeks where thin.

What is RoboCat?

A published self-improving generalist agent for robotic manipulation (arXiv 2306.11706) from Google DeepMind. It is a visual goal-conditioned decision transformer that adapts to new tasks with 100–1000 demonstrations. It is third-party research.

Why is it called RoboCat?

The paper names the agent RoboCat, and the DeepMind post explains that it is based on Gato, the Spanish word for cat.

How does RoboCat improve itself?

A fine-tuned spin-off agent practises a new task and generates more training data, which is added to the dataset used to train the next version of RoboCat. The paper says its experiments show one successful iteration.

Is RoboCat open source?

Public weights and code are Not stated in the paper or the DeepMind post, and the paper says details of one robot hand will be released in the near future.

Does Firsthand use RoboCat, and can I request a custom dataset?

Firsthand is not affiliated with the RoboCat authors and claims no use, training or compatibility. Goal images, reward labels, success or failure labels and rollouts are not Firsthand deliverables. You can request a custom dataset.