What Is Embodied Chain-of-Thought Reasoning?

ECoT is introduced in Robotic Control via Embodied Chain-of-Thought Reasoning (arXiv 2407.08693) by Michał Zawalski, William Chen, Karl Pertsch, Oier Mees, Chelsea Finn and Sergey Levine, with an official project page, GitHub repository and Hugging Face models. The authors start from a limitation of learned robot policies: they struggle to generalise outside their training data. Vision-language-action models (VLAs) help by building on large pre-trained vision-language models, yet the paper says they typically learn a direct mapping from observations to actions without any intermediate reasoning. ECoT asks whether a policy can improve by reasoning about a task before acting. This page is a definition, not a guide to running the code.

ECoT is third-party research. Firsthand is not affiliated with its authors or their institutions, and makes no claim that ECoT or any other named model was trained, fine-tuned or evaluated on Firsthand data. The site states custom egocentric collection to a written spec, fast and on demand, delivering first-person human video with 3D hand pose at 21 joints per hand and a per-joint visibility flag, calibrated stereo depth and 6-DoF IMUs at 200 Hz. The only language data tied to Firsthand capture is one task instruction per episode. Firsthand captures first-person human video, not robot or teleoperation data.

Per the paper, VLAs are pre-trained vision-language models fine-tuned to produce robot actions. The authors ask whether the step-by-step reasoning that language models use can be brought into robot control, so that a policy reasons about a given task before acting.

They name two obstacles. Simply prompting current open-source VLAs to think step-by-step is significantly less effective, because the training examples available to them are relatively simple. And purely semantic reasoning about sub-tasks is insufficient, because a robot policy has to ground its reasoning in sensory observations and the robot state. ECoT therefore trains the model to produce reasoning text and then the action, so the policy has to “think carefully” and “look carefully”, in the paper’s words.

The paper defines the chain as a fixed sequence of text outputs produced before each action. The model first rephrases the task instruction (TASK) and predicts a high-level plan (PLAN). It then names the sub-task to execute at the present step (SUBTASK) and predicts a low-level language command such as move left or move up (MOVE). Finally it predicts the pixel position of the robot end effector (GRIPPER) and the names and bounding-box pixel coordinates of all objects in the scene (OBJECTS).

The paper reports that adding these grounded steps improves on a Naive CoT ablation that reasons only about high-level language features such as sub-task plans. The authors add that they did not exhaustively explore other reasoning tasks or orderings.

The paper calls direct human annotation the gold standard for reasoning chains, but impractical for large robot datasets that consist of millions of individual transitions. The authors instead label an existing robot dataset after the fact. A vision-language model writes a scene description, an object detector finds bounding boxes, movement primitives are computed from the robot proprioception, and a large language model assembles the final reasoning chain.

They ran this on the complete Bridge v2 dataset, which the paper says has more than 2.5M transitions, over 7 days, and describe Bridge V2 as a teleoperated dataset of 60k demonstrations. The result is a set of observation-reasoning-action tuples. The reasoning labels are generated by models, not written by people, and they are derived from robot state, so the method starts from robot demonstrations.

The abstract reports that ECoT increases the absolute success rate of OpenVLA by 28% across challenging generalization tasks, without any additional robot training data. The project page says the evaluation uses 14 tasks covering novel objects, instructions and spatial relations, and the paper reports 314 trials per approach on a 6-DoF WidowX arm with a single third-person camera. It compares ECoT with Octo, with OpenVLA trained on the same Bridge data, and with RT-2-X, and reports ECoT with the highest aggregate success rate in both camera-view settings it tests.

The authors also report that inspecting the reasoning chain helps diagnose failures, and that a human can correct the policy with natural-language feedback that edits the chain, raising success rate by 48% on the most challenging tasks. These are the authors’ own evaluations on one robot setup and task suite. The paper cautions that reading reasoning chains is not a bullet-proof way to interpret failures, because the model could state a plan and still deviate from it.

The paper says the extra reasoning tokens can significantly reduce the achievable control frequency, because the number of tokens predicted per timestep rises from 7 for OpenVLA to 350 for ECoT. Its remedy is to keep parts of the chain, such as the high-level plan or the current sub-task, fixed for several steps, either synchronously, re-predicting the high-level reasoning every N steps, or asynchronously, with one policy instance updating the high-level chain while a second predicts low-level reasoning and actions.

The performance and speed trade-off sits in a later section of the paper that was not read for this page, so no control-frequency figures are given here.

The official GitHub repository, built on top of OpenVLA, is listed with an MIT licence for its code. The README releases two models, ecot-openvla-7b-bridge and ecot-openvla-7b-oxe, plus a dataset of the embodied features and reasonings for Bridge.

The README is explicit that the pretrained models may inherit restrictions from their base models. Both are derived from Llama-2 and are subject to the Llama Community License. Read that text before a commercial program relies on them.

A language-conditioned policy takes a natural-language instruction as an input. ECoT keeps that input and adds generated reasoning text between the instruction and the action. In the paper’s pipeline, each episode contributes a task instruction, and the plan, sub-tasks, movements and object boxes are generated from models around it. For data buyers the point is that the reasoning in ECoT is derived, not captured.

ECoT is built from robot demonstrations: camera images, a task instruction, robot state and robot actions, plus generated reasoning labels. Firsthand captures first-person human video, not robot or teleoperation data, so it does not deliver the robot actions or proprioception that pipeline consumes. The only language data tied to Firsthand capture is one task instruction per episode, and nothing further: no reasoning chains, narration, captions or sub-task language. Whether human egocentric data would help an ECoT-style policy is Not stated in the sources read, and this page draws no conclusion about it.

What the site does state is custom egocentric collection, fast and on demand, to a written spec. Confirm the spec on a Spec Pilot, a small fast first batch. Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks. Billing is per validated hour, and delivery is in RLDS, LeRobot, WebDataset or HDF5.

What is embodied chain-of-thought reasoning?

A method that trains a VLA to produce intermediate reasoning text, covering a plan, the current sub-task, a motion, and grounded features such as object bounding boxes and end-effector position, before it predicts each robot action. It was introduced by Zawalski and colleagues in 2024.

What does ECoT stand for?

Embodied Chain-of-Thought Reasoning. The paper uses ECoT for the method and for the policies trained with it.

How much does ECoT improve OpenVLA?

The abstract reports a 28% increase in absolute success rate across challenging generalization tasks, without any additional robot training data. That is the authors’ own evaluation on one robot setup and task suite.

Does ECoT slow a policy down?

The paper says the extra reasoning tokens can significantly reduce the achievable control frequency, with tokens per timestep rising from 7 for OpenVLA to 350 for ECoT. It proposes keeping parts of the chain fixed for several steps to recover speed.

Is ECoT open source, and what is the licence?

The official GitHub repository is listed with an MIT licence for code. The README says the released models are derived from Llama-2 and subject to the Llama Community License, so read the licence text before a commercial program relies on them.

Did ECoT use Firsthand data, and can I request a custom dataset?

No. Firsthand is not affiliated with the ECoT authors and makes no claim that ECoT or any model was trained on Firsthand data. You can request a custom egocentric dataset: write the tasks, fields and format into a spec, confirm it on a Spec Pilot, and billing is per validated hour. The only language data is one task instruction per episode, and Firsthand captures first-person human video, not robot or teleoperation data.