What Is VoxPoser?

VoxPoser is introduced in VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models (arXiv 2307.05973) by Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu and Li Fei-Fei. The project page lists Stanford University and the University of Illinois Urbana-Champaign as affiliations and CoRL 2023 (Oral) as the venue. The abstract states the aim: to synthesize robot trajectories, a dense sequence of 6-DoF end-effector waypoints, for many manipulation tasks given an open set of instructions and an open set of objects, without relying on pre-defined motion primitives.

Firsthand is not affiliated with the VoxPoser authors and makes no claim that any model, VoxPoser included, was trained on Firsthand data. Firsthand captures human egocentric video, not robot data: custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license. 3D value maps, voxel maps, camera calibration files, multi-view rigs, 3D labels, reward labels, success or failure labels, rollouts, planning annotations and language annotations beyond one task instruction per episode are not Firsthand deliverables.

Per the abstract, large language models are good at inferring affordances and constraints from a free-form instruction. By using their code-writing ability, they can interact with a vision-language model to compose 3D value maps that ground that knowledge in the agent’s observation space.

The composed maps then feed a model-based planning framework that synthesizes closed-loop trajectories zero-shot, with robustness to dynamic perturbations. The paper’s example is “open the top drawer and watch out for the vase”: the drawer handle region gets high values and the area around the vase gets low values.

The paper defines a voxel value map as a 3D grid over the robot’s observation space that guides an “entity of interest”, such as the end-effector, an object or an object part. It names five types: affordance, avoidance, end-effector velocity, end-effector rotation and gripper action maps.

The LLM fills the maps by writing Python that calls perception APIs and NumPy array operations. Per the paper, the perception stack uses the open-vocabulary detector OWL-ViT, Segment Anything and the video tracker XMEM, and the language model is GPT-4 prompted with example queries. The input is an RGB-D observation plus a language instruction.

The authors use simple zeroth-order optimization inside model predictive control. Only the affordance and avoidance maps drive the planner; the other maps set rotation, velocity and gripper action at each waypoint. After a 6-DoF trajectory is synthesized, the first waypoint is executed and a new trajectory is replanned at 5 Hz. Waypoints are executed by an Operational Space Controller.

Because the generated code stays the same throughout a sub-task, its output can be cached, which makes closed-loop replanning possible. The paper also shows that zero-shot trajectories can seed learning of a dynamics model from online interaction for contact-rich tasks such as doors, windows and fridges in simulation.

Per the paper, the real-world setup is a Franka Emika Panda arm on a tabletop with two Azure Kinect RGB-D cameras returning observations at 20 Hz. The authors evaluate five real-world tasks and a simulated suite of 13 tasks. These are the authors’ own setups, and this page repeats none of their success rates.

The authors list four limits: VoxPoser relies on external perception modules, a general-purpose dynamics model is still needed for contact-rich tasks, the motion planner considers only end-effector trajectories, and the LLM prompts need manual engineering. The paper states an open-source implementation, linked in the sources below.

The comparison table below sets VoxPoser beside four site entries. The paper cites PaLM-E in its related work and says multi-modal LLMs of that kind could be translated into VoxPoser for direct visual grounding. It does not mention RoboBrain, GeoVLA or 3D-VLA, so those rows say Not stated.

In short, VoxPoser composes value maps from LLM-written code and plans without training a new policy, while the other four entries describe trained models. This page ranks no models.

In its zero-shot form, the paper says VoxPoser needs no additional training data for each task or for the LLM. It contrasts this with robotic data that are “often of limited amount or variability”, and relies on pre-trained language and vision-language models instead.

Firsthand states human egocentric capture with stereo-derived depth, 21-joint hand pose and one task instruction per episode. Whether those streams suit VoxPoser is Not stated, and no Firsthand data is claimed to have trained or been used by VoxPoser.

When a program needs real human egocentric demonstrations that public data does not cover. Firsthand does not build or train models. Confirm the fields on a Spec Pilot, a small fast first batch; the median from signed spec to first validated batch is 48 hours where coverage is deep, three to four weeks where thin.

What is VoxPoser?

A published method (arXiv 2307.05973) in which a language model writes code that composes 3D value maps through a vision-language model, and a motion planner turns the maps into robot trajectories without additional training. It is third-party research.

What are composable 3D value maps?

Per the paper, voxel grids over the robot’s observation space that encode affordances and constraints. Types are affordance, avoidance, end-effector velocity, end-effector rotation and gripper action maps, composed by LLM-written code.

Does VoxPoser need robot training data?

Per the paper, not for the zero-shot synthesis: it needs no additional training data for each task or for the LLM. The authors also show optional online learning of a dynamics model for contact-rich tasks.

Are the VoxPoser code and prompts available?

The paper states an open-source implementation, and the project page links the authors’ materials. The GitHub repository is linked in the sources below. Check its licence before relying on it.

Does Firsthand use VoxPoser, and can I request a custom dataset?

Firsthand is not affiliated with the VoxPoser authors and claims no use, training or compatibility. 3D value maps, voxel maps, camera calibration files, rollouts, reward labels and success or failure labels are not Firsthand deliverables. You can request a custom dataset.