What Is Inner Monologue?

Inner Monologue is introduced in Inner Monologue: Embodied Reasoning through Planning with Language Models (arXiv 2207.05608) by Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman and Brian Ichter, submitted on 12 July 2022. The abstract says embodied problems require an agent to understand the skills available, how they influence the world and how changes to the world map back to language. A planner must decide not just what skills to do but how and when to do them, and those answers change over time in response to the agent’s own choices.

Firsthand is not affiliated with the Inner Monologue authors and makes no claim that any model, Inner Monologue included, was trained on Firsthand data. Firsthand captures human egocentric video, not robot data: custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license. Language feedback, scene descriptions, success detection labels, plans, robot trajectories, teleoperation data, action labels in robot action spaces, rollouts and language annotations beyond one task instruction per episode are not Firsthand deliverables.

Per the abstract, the authors investigate to what extent language models used in embodied contexts can reason over sources of feedback provided through natural language, without any additional training. They propose that by leveraging environment feedback, the model forms an inner monologue that lets it process and plan more richly in robotic control scenarios.

The project page explains the gap it addresses. Earlier planning work with language models was one-directional: the model influences the agent and the environment, but no feedback is routed back to it. That matters most when an intermediate action fails, because the model is not told. Inner Monologue continually adds information from several feedback sources into the language model prompts.

The abstract names success detection, scene description and human interaction, and the project page adds object recognition. The page says the studies focus on three types. Passive scene description is feedback provided consistently in a structured form, such as object recognition results.

Active scene description covers free-form questions the model may ask and the unstructured answers given by a learned model, for example a visual question answering model, or by a person, and it can also inject human preference during plan generation. Success detection is binary feedback on whether the last action succeeded, which the page calls useful in long-horizon settings.

The abstract reports three domains: simulated and real tabletop rearrangement tasks, and long-horizon mobile manipulation tasks in a real kitchen environment. It states that closed-loop language feedback significantly improves high-level instruction completion across them. The sources read give no scores in the abstract or the project page, so this page repeats none.

The project page describes a simulated tabletop with instructions such as “put all blocks in bowls with matching colors” and “stack all the blocks,” where scene description tracks achieved sub-tasks and the model writes chain-of-thought text about what remains. In the real tabletop it uses object recognition and success detection while perception may be subject to occlusions. In the kitchen, it builds on SayCan, uses success detection, and forces failures with adversarial human interventions to show replanning, for example on “put a coke in the top drawer.”

The project page lists behaviors it says go beyond the prompted examples when the model is informed with environment feedback. These include adapting to new instructions that change the goal mid-task, proposing alternative goals when a goal becomes infeasible, and following instructions given in another language, shown with a Chinese instruction re-narrated as an English goal.

The page also lists interactive scene understanding, where the model answers questions about the scene using past actions and feedback as context, plus robustness to the order of feedback and to typos in human instructions. These are the authors’ qualitative demonstrations, and this page ranks no methods.

The comparison table below sets Inner Monologue beside the SayCan and Code as Policies entries. The sources read for Inner Monologue say the kitchen experiments build on SayCan. The SayCan entry states that PaLM-SayCan only receives environmental feedback through value functions at the current decision step, and points to follow-up work that routes success detectors, scene descriptors or human feedback through an inner monologue.

The sources read for Inner Monologue do not discuss Code as Policies, so that row says Not stated. In short, SayCan picks a skill by combining language model and value-function scores, Code as Policies has a model write policy code, and Inner Monologue is about feeding language feedback back into the planner’s prompt. This page ranks no methods.

Per the sources read, Inner Monologue needs a language model prompted to plan, skills or a primitive policy the plan can call, and sources of language feedback such as object recognition, success detection or a person. The project page says it does not depend on a specific language model or type of grounding feedback. The sources read state no training dataset requirement, so this page does not say which data it needs.

Firsthand states human egocentric capture with stereo-derived depth, 21-joint hand pose and one task instruction per episode, and no language feedback, scene descriptions, success detection labels or plans. Whether those streams suit an Inner Monologue-style system is Not stated, and no Firsthand data is claimed to have trained or been used by it.

When a program needs real human egocentric demonstrations that public data does not cover. Firsthand does not build or train models. Confirm the fields on a Spec Pilot, a small fast first batch; the median from signed spec to first validated batch is 48 hours where coverage is deep, three to four weeks where thin.

What is Inner Monologue?

A published method (arXiv 2207.05608) in which a language model that plans for a robot reads environment feedback in natural language, such as success detection and scene description, and replans from it, without additional training. It is third-party research.

What is the inner monologue itself?

Per the project page, it is formed by continually adding information from sources of feedback into the language model prompts, so the planner sees what happened, not only what it asked for.

What feedback does it use?

The paper studies success detection, passive scene description, active scene description and human interaction. Passive scene description can be object recognition results; active scene description answers free-form questions from a learned model or a person.

What results do the authors report?

The abstract says closed-loop language feedback significantly improves high-level instruction completion on three domains: simulated and real tabletop rearrangement, and long-horizon mobile manipulation in a real kitchen. The sources read state no scores, so none are repeated here.

Does Firsthand use Inner Monologue, and can I request a custom dataset?

Firsthand is not affiliated with the Inner Monologue authors and claims no use, training or compatibility. Language feedback, scene descriptions, success detection labels, plans, robot trajectories, action labels and rollouts are not Firsthand deliverables. You can request a custom dataset.