What Is SayCan?

SayCan is introduced in Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (arXiv 2204.01691), a long author list led by Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar and Omar Cortes. The abstract states the problem: language models encode semantic knowledge useful for high-level, temporally extended instructions, but lack real-world experience, so their advice may not apply to a particular robot in a particular environment. Asking a language model how to clean a spill may give a reasonable narrative that the robot cannot carry out.

Firsthand is not affiliated with the SayCan authors and makes no claim that any model, SayCan included, was trained on Firsthand data. Firsthand captures human egocentric video, not robot data: custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license. Robot trajectories, teleoperation data, value functions, affordance labels, action labels in robot action spaces, success or failure labels, rollouts and language annotations beyond one task instruction per episode are not Firsthand deliverables.

Per the paper, the robot acts as the language model’s “hands and eyes,” while the language model supplies high-level knowledge about the procedure. Pretrained low-level skills constrain the model to propose natural language actions that are both feasible and contextually appropriate. Value functions associated with those skills connect the language model’s knowledge to the physical environment.

The project page gives the loop. Given a high-level instruction, SayCan combines the probability from the language model, that a skill is useful for the instruction, with the probability from a value function, that the skill will execute successfully. The skill with the highest combined score is chosen and executed, appended to the robot’s response, and the models are queried again until the step is to terminate.

The project page says each skill has an accompanying affordance function that quantifies how likely the skill is to succeed from the current state, such as a learned value function. Its value weights the skill’s language-model likelihood. This is a robot-side use of the word affordance: a score for whether a skill can work now, not a map of where to grasp an object.

The authors frame planning as a dialog. A user gives an instruction such as “How would you bring me a coke can?” and the model answers with an explicit sequence of steps ending in “Done.” They show the affordances overriding the language model: it favored picking up a sponge, the affordances knew that was not possible, and “find a sponge” was chosen instead.

The abstract says the method was evaluated on real-world robotic tasks on a mobile manipulator, where it completed long-horizon, abstract, natural language instructions. The project page describes two scenes, an office kitchen and a mock office kitchen, with 101 tasks specified by natural language instructions.

For the later version that uses the PaLM language model, the project page reports a plan success rate of 84% and an execution success rate of 74% over those 101 tasks, which it states are 14% and 13% higher than the initial release. These are the authors’ figures from their own setup; read the paper for how they were measured. This page ranks no methods.

The August 2022 update, per the arXiv comments and project page, added PaLM results and new capabilities: drawer manipulation, chain of thought prompting and multilingual instructions. New skills are added by listing them as options for the language model, supplying their value functions and adding a prompt example. The page reports almost no drop in planning success when queries move from English to Chinese, French and Spanish.

The project page lists a SayCan dataset v0 that maps natural language user instructions to one possible solution, available under CC BY 4.0, and an open-source version of SayCan on a simulated tabletop environment. This page links neither directly; use the official project page.

The project page states that PaLM-SayCan only receives environmental feedback through value functions at the current decision step. If a skill fails or the environment changes, the necessary feedback may not be available. It points to follow-up work that builds closed-loop planning on it using environment feedback from success detectors, scene descriptors or human feedback, through an inner monologue.

The sources read do not state SayCan’s skill count, how the skills and value functions were trained, or the robot’s hardware beyond a mobile manipulator, so this page states none of them.

The comparison table below sets SayCan beside four site entries, using only the sources read for SayCan. The project page mentions the PaLM language model inside PaLM-SayCan, but the sources read do not discuss PaLM-E or VoxPoser, which are separate papers, so those rows say Not stated.

In short, SayCan is a planner that uses a language model to choose among pretrained skills, a language-conditioned policy takes the instruction directly, and an affordance is the underlying idea of what an object or state allows. This page ranks no models.

Per the sources read, SayCan needs pretrained low-level skills with associated value functions, plus a language model prompted to plan. How those skills and value functions were trained is Not stated in the sources read, so this page does not say which data they need.

Firsthand states human egocentric capture with stereo-derived depth, 21-joint hand pose and one task instruction per episode, and no robot skills, plans or value functions. Whether those streams suit SayCan is Not stated, and no Firsthand data is claimed to have trained or been used by SayCan.

When a program needs real human egocentric demonstrations that public data does not cover. Firsthand does not build or train models. Confirm the fields on a Spec Pilot, a small fast first batch; the median from signed spec to first validated batch is 48 hours where coverage is deep, three to four weeks where thin.

What is SayCan?

A published robot planning method (arXiv 2204.01691) in which a language model scores which pretrained skill helps with an instruction and a value function scores whether that skill can succeed from the current state. The combined score picks the next skill. It is third-party research.

What does Do As I Can, Not As I Say mean?

It is the paper title. The idea is that the robot’s feasible skills, scored by value functions, constrain what the language model proposes, so the plan reflects what the robot can do rather than only what the model says.

What are affordance value functions in SayCan?

Per the project page, each skill has an affordance function that quantifies how likely it is to succeed from the current state, such as a learned value function. Its value weights the language model’s likelihood for that skill.

What results do the SayCan authors report?

The project page reports 84% plan success and 74% execution success for PaLM-SayCan on 101 natural language tasks in an office kitchen and a mock office kitchen. These are the authors’ figures; read the paper for how they were measured.

Does Firsthand use SayCan, and can I request a custom dataset?

Firsthand is not affiliated with the SayCan authors and claims no use, training or compatibility. Robot trajectories, value functions, affordance labels, action labels, rollouts and success or failure labels are not Firsthand deliverables. You can request a custom dataset.