What Is RoboPoint?

RoboPoint is introduced in RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics (arXiv 2406.10721) by Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Arsalan Mousavian and Dieter Fox, submitted on 15 June 2024. The project page lists CoRL 2024 and the affiliations University of Washington, NVIDIA, the Allen Institute for AI and Universidad Católica San Pablo. The abstract starts from a gap: robots must plan precise action points, but vision-language models struggle to articulate robot actions precisely in language. RoboPoint answers by pointing: given an image and an instruction, it predicts the image points where the robot should act.

Firsthand is not affiliated with the RoboPoint authors and makes no claim that any model, RoboPoint included, was trained on Firsthand data. Firsthand captures human egocentric video, not robot data: custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license. Point labels, affordance annotations, keypoint or mask labels, synthetic data, robot trajectories, teleoperation data and language annotations beyond one task instruction per episode are not Firsthand deliverables.

Per the paper, RoboPoint is a vision-language model instruction-tuned to point. It has two features that set it apart from other robotics VLMs: a point-based action space and a scalable data pipeline. The model fine-tunes on spatial affordance prediction, the task of pointing at where to act, so an answer is a set of image coordinates that satisfy the relations in a language prompt, such as “place the cup next to the plate.”

The authors argue this is more precise than fuzzy language actions or bounding boxes, which can include clutter. They say the predicted points can be read as waypoints for navigation, contact points for grasping or region proposals for placement, which lets one model serve several tasks.

The project page describes the deployment path. RoboPoint predicts 2D action points from an image and an instruction, the points are projected into 3D using a depth map, and the robot then navigates to those 3D targets with a motion planner. In the paper’s manipulation setup, the end-effector pose is computed from the 3D points plus an offset and a motion planner executes the trajectory.

The paper says this removes the need for pre-defined action primitives, external object detectors or iterative visual prompting. A depth map is therefore an input to the execution step, not something the model predicts. The authors also list a limit: RoboPoint gives no confidence estimates for its points, and the number of output points is not controllable.

The abstract says the authors built an automatic synthetic data generation pipeline, and that the method requires no real-world data collection or human demonstration. The pipeline renders RGB images from procedurally generated 3D scenes, computes spatial relations from the camera’s perspective, and samples points within object masks and object-surface intersections to make instruction-point pairs. The paper says around 660K image-relation pairs come from 10K scenes, and that free-space affordances are made by removing the target object, re-rendering and sampling points on the supporting surface.

The fine-tuning mix has four sources, per the paper’s Table 1: object reference (347K) and free space reference (320K) from the synthetic pipeline, plus VQA data (665K) and object detection data from LVIS (100K). The model is instruction-tuned from Vicuna-v1.5-13B, per the paper. The abstract’s “no real-world data collection” therefore means no new robot data, since real-image VQA and detection data are part of the mix.

The abstract states that RoboPoint outperforms GPT-4o and the visual prompting technique PIVOT by 21.8% in accuracy of predicting spatial affordance and by 30.5% in the success rate of downstream tasks. These are the authors’ numbers on their own benchmarks and setups; Firsthand does not reproduce them and quotes no others.

For evaluation the paper uses an object reference subset of RoboRefIt and Where2Place, a real-world benchmark of 100 images it collected for free space reference. Downstream, it shows real-world manipulation across three environments with seven tasks, navigation in simulated room scenes, and an augmented reality assistant, since the co-training with VQA data keeps conversational ability.

The official repository, wentaoyuan/RoboPoint, is released under the Apache License 2.0. Its README describes a model zoo of several checkpoints, a Gradio demo, the training data mix, training scripts and an evaluation script for Where2Place. It says the data mix and the Where2Place benchmark are hosted on Hugging Face and builds on the LLaVA codebase.

The comparison table below sets RoboPoint beside the Affordance, VoxPoser and vision-language-action model entries. The paper names affordance as the functions of an object, meaning in what ways it can be manipulated, and says it can be represented as part segmentation, dense descriptors or keypoints. RoboPoint uses 2D keypoints because they convert readily to language.

The paper lists VoxPoser as related work that generates 3D value maps, and says that VoxPoser, PIVOT and MOKA still rely on external models to detect objects relevant to the task, unlike RoboPoint. The sources read do not describe RoboPoint as a vision-language-action model, so that row says Not stated. This page ranks no methods.

Per the sources read, RoboPoint’s fine-tuning data is largely synthetic image-and-point pairs with templated language, plus public VQA and detection data, and the paper says that needs no human demonstrations. Whether human egocentric video would help a pointing model is Not stated in the sources read.

Firsthand states human egocentric capture with stereo-derived depth, 21-joint hand pose and one task instruction per episode, and no point labels, affordance annotations, masks or synthetic data. No Firsthand data is claimed to have trained or been used by RoboPoint.

When a program needs real human egocentric demonstrations that public or synthetic data does not cover. Firsthand does not build or train models. Confirm the fields on a Spec Pilot, a small fast first batch; the median from signed spec to first validated batch is 48 hours where coverage is deep, three to four weeks where thin.

What is RoboPoint?

A published vision-language model (arXiv 2406.10721) that predicts image keypoint affordances, meaning 2D points showing where a robot should act, from a language instruction. It is third-party research.

How does RoboPoint turn points into robot actions?

Per the project page, the predicted 2D points are projected into 3D using a depth map, and the robot then navigates to those targets with a motion planner.

What data was RoboPoint trained on?

Per the paper, a mix of synthetic object and free space reference data from a procedural pipeline, plus VQA data and LVIS object detection data. The abstract says the method requires no real-world data collection or human demonstration.

What results do the authors report?

The abstract says RoboPoint beats GPT-4o and PIVOT by 21.8% in accuracy of predicting spatial affordance and by 30.5% in success rate of downstream tasks. These are the authors’ numbers on their own setups.

Does Firsthand use RoboPoint, and can I request a custom dataset?

Firsthand is not affiliated with the RoboPoint authors and claims no use, training or compatibility. Point labels, affordance annotations, masks, synthetic data, robot trajectories and rollouts are not Firsthand deliverables. You can request a custom dataset.