What Is PaLM-E?

PaLM-E is introduced in PaLM-E: An Embodied Multimodal Language Model (arXiv 2303.03378) by Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Pete Florence and colleagues at Robotics at Google, TU Berlin and Google Research, first submitted on 6 March 2023. The authors name it PaLM-E because it uses PaLM as the pre-trained language model and makes it Embodied. Read: the arXiv abstract, the full paper text and the official project page. No official GitHub repository is linked from them.

Firsthand is not affiliated with the PaLM-E authors and makes no claim that any model, PaLM-E included, was trained on Firsthand data. Firsthand captures human egocentric video, not robot data: custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license. Object state vectors, object instance masks, 3D scene representations, planning annotations, reward labels, success or failure labels, rollouts and language annotations beyond one task instruction per episode are not Firsthand deliverables.

Per the paper, observations are encoded into sequences of vectors with the same dimension as the language token embeddings, then interleaved with text to form “multi-modal sentences”. PaLM-E is a decoder-only model that generates text autoregressively.

The authors study three kinds of encoder: state estimation vectors, Vision Transformers (ViT) for 2D image features, and the 3D-aware Object Scene Representation Transformer (OSRT), plus object-centric variants that tokenize individual objects.

The output is text: an answer to a question, or a sequence of decisions a robot should carry out. For planning, the paper assumes low-level policies that perform skills from a small vocabulary. PaLM-E acts as a high-level policy that sequences them inside a control loop, replanning from new observations.

On the Language-Table setup the authors state that PaLM-E outputs language subgoals at 1 Hz to policies that output low-level actions at 5 Hz. In the mobile manipulation domain the low-level policies are from RT-1.

The authors report a “full mixture” dominated by internet-scale vision-language data, in which embodied data is 8.9% of samples. Their robot environments are task and motion planning (TAMP) in simulation, a Language-Table tabletop pushing setup in simulation and on a real robot, and a real kitchen mobile manipulation domain trained on runs from earlier SayCan work.

For Language-Table, the paper describes labelers entering a short-horizon command every four seconds. All of this is the authors’ own robot data, not egocentric human video.

These are the authors’ results on their own setups; Firsthand does not reproduce them and ranks no models. They train three sizes, listed below, and state that PaLM-E-562B reaches state-of-the-art performance on the OK-VQA benchmark.

They also report positive transfer: training on the mixture beat training on single-domain data, including from 10 to 80 demonstrations per Language-Table task and 320 examples per TAMP planning task. The real kitchen evaluation is described as qualitative.

The paper contrasts PaLM-E with models that predict actions directly: PaLM-E generates high-level instructions as text and relies on separate low-level policies. The site’s RT-2 entry states that RT-2 writes actions as text tokens and that one of its variants, RT-2-PaLM-E, is built from a 12B parameter model.

Whether PaLM-E counts as a VLA model or a robot foundation model is a classification the sources read do not make, so it is Not stated here.

The authors report that off-the-shelf vision-language models trained on general question answering and captioning do not solve their embodied reasoning tasks directly, and that freezing the language model preserved its language skills but occasionally struggled on robotics tasks. Training the whole model loses language ability unless the model is scaled up.

PaLM-E depends on low-level policies to act. Public weights and code are Not stated in the sources read.

The paper trains on expert data from its three robot domains. Its Figure 2 shows a zero-shot question answering example on temporally annotated egocentric video, a qualitative demonstration by the authors rather than a training set.

Firsthand states human egocentric capture with stereo-derived depth, 21-joint hand pose and one task instruction per episode. Whether those streams match PaLM-E’s inputs is Not stated.

When a program needs real human egocentric demonstrations that public data does not cover. Firsthand does not build or train models. Confirm the fields on a Spec Pilot, a small fast first batch; the median from signed spec to first validated batch is 48 hours where coverage is deep, three to four weeks where thin.

What is PaLM-E?

A published embodied multimodal language model (arXiv 2303.03378) that injects images, state estimates and other sensor inputs into a pre-trained PaLM language model and generates text, including robot plans. It is third-party research.

What does the E in PaLM-E stand for?

Embodied. The authors state that they call the model PaLM-E because they use PaLM as the pre-trained language model and make it Embodied.

Is PaLM-E a VLA model?

The paper contrasts PaLM-E with models that output actions directly. PaLM-E generates high-level instructions as text for low-level policies to carry out. Calling it a VLA model is a classification the sources read do not make.

Is PaLM-E open source?

Public weights and code are Not stated in the paper or the project page, which show videos and example completions. No official GitHub repository is linked from them.

Does Firsthand use PaLM-E, and can I request a custom dataset?

Firsthand is not affiliated with the PaLM-E authors and claims no use, training or compatibility. Object masks, planning annotations, reward labels and rollouts are not Firsthand deliverables. You can request a custom dataset.