What Is Gato (A Generalist Agent)?

Gato is introduced in A Generalist Agent (arXiv 2205.06175) by Scott Reed, Konrad Zolna, Emilio Parisotto and colleagues, published in Transactions on Machine Learning Research in November 2022. The abstract says “the agent, which we refer to as Gato”, which names it. Read: the arXiv abstract page and the official Google DeepMind post. The full paper text was not read for this page.

Firsthand is not affiliated with the Gato authors and makes no claim that any model, Gato included, was trained on Firsthand data. Firsthand captures human egocentric video, not robot data: custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license. Camera calibration files, multi-view rigs, 3D labels, reward labels, success or failure labels, rollouts, latent action labels, planning annotations and language annotations beyond one task instruction per episode are not Firsthand deliverables.

Per the DeepMind post, data from different tasks and modalities is serialised into a flat sequence of tokens, batched and processed by a transformer similar to a large language model. The loss is masked so Gato predicts only action and text targets.

At deployment a prompt, such as a demonstration, is tokenised to form the initial sequence. The environment’s observation is tokenised and appended, Gato samples the action vector autoregressively one token at a time, and the decoded action is sent back to the environment. The post says it always sees all previous observations and actions within a context window of 1024 tokens.

The abstract says the same network with the same weights can play Atari, caption images, chat and stack blocks with a real robot arm, deciding from its context whether to output text, joint torques, button presses or other tokens. That breadth is why the authors call it a multi-modal, multi-task, multi-embodiment generalist policy.

The comparison table below sets Gato beside four site entries. In short, RT-1 is a robot control model, PaLM-E emits text plans, RoboCat builds on the Gato architecture per its own paper, and cross-embodiment learning is the training idea Gato’s abstract names.

This page ranks no models and repeats no benchmark scores, success rates, parameter counts or task counts, because none were read as stated values in the sources.

The DeepMind post says Gato is trained on a large number of datasets of agent experience in simulated and real-world environments, plus natural language and image datasets. Whether any of that is egocentric human video is Not stated in the sources read.

Firsthand states human egocentric capture with stereo-derived depth, 21-joint hand pose and one task instruction per episode. Whether those streams match Gato’s inputs is Not stated, and no Firsthand data is claimed to have trained it.

When a program needs real human egocentric demonstrations that public data does not cover. Firsthand does not build or train models. Confirm the fields on a Spec Pilot, a small fast first batch; the median from signed spec to first validated batch is 48 hours where coverage is deep, three to four weeks where thin.

What is Gato (A Generalist Agent)?

A published generalist agent (arXiv 2205.06175) from Google DeepMind: one transformer network with the same weights that can play Atari, caption images, chat and stack blocks with a real robot arm. It is third-party research.

What does multi-embodiment mean for Gato?

The abstract calls Gato a multi-modal, multi-task, multi-embodiment generalist policy, meaning one network decides from context whether to output text, joint torques, button presses or other tokens.

How does Gato choose actions?

Per the DeepMind post, it tokenises a prompt and the latest observation, then samples the action vector autoregressively, one token at a time, within a context window of 1024 tokens.

Are the Gato weights or code available?

Not stated in the abstract or the DeepMind post, which are the sources read. This page links no repository; check the paper before assuming any release.

Does Firsthand use Gato, and can I request a custom dataset?

Firsthand is not affiliated with the Gato authors and claims no use, training or compatibility. Camera calibration files, rollouts, reward labels and success or failure labels are not Firsthand deliverables. You can request a custom dataset.