What Is Code as Policies?

Code as Policies is introduced in Code as Policies: Language Model Programs for Embodied Control (arXiv 2209.07753) by Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence and Andy Zeng. The abstract starts from an observation: large language models trained on code completion can synthesize simple Python programs from docstrings, and the authors find these code-writing models can be re-purposed to write robot policy code from natural language commands. The paper calls the generated programs language model programs, or LMPs, and presents code as policies as a robot-centric formulation of them.

Firsthand is not affiliated with the Code as Policies authors and makes no claim that any model, Code as Policies included, was trained on Firsthand data. Firsthand captures human egocentric video, not robot data: custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license. Generated code, policy programs, perception API outputs, robot trajectories, teleoperation data, action labels in robot action spaces, success or failure labels, rollouts and language annotations beyond one task instruction per episode are not Firsthand deliverables.

Per the abstract, policy code can express functions or feedback loops that process perception outputs, for example from object detectors, and parameterize control primitive APIs. The language model is shown several example language commands, formatted as comments, followed by the corresponding policy code. This is few-shot prompting. Given a new command, the model re-composes API calls to generate new policy code on its own.

The authors say that by chaining classic logic structures and referencing third-party libraries such as NumPy and Shapely to do arithmetic, the generated policies can exhibit spatial-geometric reasoning, generalize to new instructions, and prescribe precise values, such as velocities, for ambiguous descriptions like “faster” depending on context. They call the last of these behavioral commonsense.

The abstract names prompting hierarchical code-gen, which recursively defines undefined functions, as central to the approach. The model writes high-level code that calls helper functions it has not yet defined, and those functions are then generated in turn. According to the authors, this can write more complex code.

The abstract also reports that this improves the state of the art to solve 39.8% of problems on the HumanEval benchmark. That figure is the authors’ own, about code generation rather than robot performance, and this page repeats no other scores.

Per the abstract, language model programs can represent reactive policies, such as impedance controllers, as well as waypoint-based policies, such as vision-based pick and place and trajectory-based control. The paper demonstrates the approach across multiple real robot platforms.

The authors’ project page groups its experiment videos and generated code into tabletop manipulation with blocks, with blocks and bowls, and with fruits, bottles and plates, plus whiteboard drawing, mobile robot navigation and mobile robot manipulation. The page also lists the prompts used in each domain and notes that different prompts specialize the language model to different functions, composed together by generated code through function calls that take natural language arguments.

The project page says its videos include voice and speech-based robot interfaces, and that long pauses between commands and responses are mostly caused by OpenAI API query times and rate limiting. Latency in those demos therefore came largely from calling a hosted language model, not from the robot.

The abstract states that code and videos are available at the project website, which this page links. The sources read for this page do not state a repository license, a model count or the robot hardware used, so this page states none of them.

The table below sets Code as Policies beside three site entries. The sources read for Code as Policies do not discuss SayCan, VoxPoser or vision-language-action models, so the right-hand column states only how Code as Policies itself works and marks any relation to them as Not stated. This page ranks no methods.

In short, Code as Policies has a language model write a program that calls perception and control APIs, a language-conditioned policy takes the instruction directly, and a vision-language-action model maps a scene and an instruction to an action.

Per the sources read, Code as Policies needs a code-writing language model, a few example command-and-code prompts, and perception and control primitive APIs that the generated code can call. The abstract describes few-shot prompting rather than training a new policy, and the sources read do not state any robot dataset requirement, so this page does not say which data it needs.

Firsthand states human egocentric capture with stereo-derived depth, 21-joint hand pose and one task instruction per episode, and no generated code, API outputs or robot actions. Whether those streams suit a Code as Policies-style system is Not stated, and no Firsthand data is claimed to have trained or been used by it.

When a program needs real human egocentric demonstrations that public data does not cover. Firsthand does not build or train models. Confirm the fields on a Spec Pilot, a small fast first batch; the median from signed spec to first validated batch is 48 hours where coverage is deep, three to four weeks where thin.

What is Code as Policies?

A published method (arXiv 2209.07753) in which a code-writing large language model, prompted with a few example commands and code, writes robot policy code from natural language commands. The code processes perception outputs and calls control primitive APIs. It is third-party research.

What are language model programs?

Per the authors, programs generated by a language model. Code as Policies is their robot-centric formulation, covering reactive policies such as impedance controllers and waypoint-based policies such as vision-based pick and place.

Does Code as Policies train a new robot policy?

The abstract describes few-shot prompting: example commands formatted as comments followed by policy code, after which the model writes new code for new commands. The sources read state no robot dataset requirement.

What result do the authors report?

The abstract says hierarchical code generation improves the state of the art to solve 39.8% of problems on the HumanEval benchmark. That is the authors’ own figure about code generation, and this page quotes no other scores.

Does Firsthand use Code as Policies, and can I request a custom dataset?

Firsthand is not affiliated with the Code as Policies authors and claims no use, training or compatibility. Generated code, policy programs, perception API outputs, robot trajectories, action labels, rollouts and success or failure labels are not Firsthand deliverables. You can request a custom dataset.