What Is MOKA?

MOKA is introduced in MOKA: Open-World Robotic Manipulation through Mark-Based Visual Prompting (arXiv 2403.03174) by Fangchen Liu, Kuan Fang, Pieter Abbeel and Sergey Levine, first submitted on 5 March 2024. The project page lists Berkeley AI Research, UC Berkeley, and the Robotics: Science and Systems (RSS) 2024 venue. The abstract starts from a problem: vision-language models offer unprecedented opportunities for open-world problems, but how to leverage them to control robots remains a grand challenge. MOKA is the authors’ approach, built on a compact point-based representation of affordance and a visual prompting technique that annotates marks on images.

Firsthand is not affiliated with the MOKA authors and makes no claim that any model, MOKA included, was trained on Firsthand data. Firsthand captures human egocentric video, not robot data: custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license. Keypoint labels, mark annotations, point labels, affordance labels, masks, synthetic data, robot trajectories, teleoperation data and language annotations beyond one task instruction per episode are not Firsthand deliverables.

Per the abstract, MOKA stands for Marking Open-world Keypoint Affordances. It employs vision-language models to solve robotic manipulation tasks specified by free-form language instructions. The central element is a compact point-based representation of affordance that bridges the model’s predictions on observed images and the robot’s actions in the physical world.

The abstract says that by prompting a pre-trained VLM, the approach uses the model’s commonsense knowledge and concept understanding, acquired from broad data sources, to predict affordances and generate motions. In the glossary sense, an affordance is what an object lets an agent do; here it is expressed as points on the image rather than as a trajectory or a value map.

The abstract proposes a visual prompting technique that annotates marks on images, converting affordance reasoning into a series of visual question-answering problems that the VLM can solve in zero-shot and few-shot manners. The project page names the marks: candidate points, grids and captions drawn on RGB images. The model is then asked, in effect, which marked point or grid cell answers each question.

The point of the marks is to give a language-and-vision model something it can name. Instead of asking the model for pixel coordinates in free text, the prompt shows labelled candidates and the model chooses among them. The project page says the VLM used is GPT-4V.

The project page describes a hierarchical visual prompting framework. At the high level, the VLM is prompted to decompose the free-form description of the task into a sequence of subtasks and to summarize the subtask information. At the low level, the VLM produces the point-based affordance representation from the marked image.

To bridge the model’s predictions on 2D images and motion in the physical world, the page introduces the point-based affordance representation: a set of keypoints and waypoints that specify the robot’s motion for a wide range of tasks. The evaluation described in the abstract covers table-top manipulation tasks including tool use, deformable body manipulation and object rearrangement. Firsthand reproduces no success rates from the paper.

The abstract says the authors further explore methods to enhance performance with robot experiences collected by MOKA, through in-context learning and policy distillation. That is robot data produced by the method running on a robot, not human demonstrations. The detail of those experiments sits in the paper body, which this entry does not summarize.

The official repository, moka-manipulation/moka, is described as the official implementation for the RSS 2024 paper and carries an MIT License. Its README says the perception pipeline is adapted from Grounded-SAM, GroundingDINO and SAM, that the robot platform is built on DROID, and that the visual-prompting inspiration comes from SoM. Running it needs an OpenAI API key, and the full pipeline command targets a Franka robot, with a note that other setups modify a policy file.

The README also describes an optional step to convert stored robot data to RLDS format, compatible with the Octo dataloader, for fine-tuning a pre-trained model. RLDS also appears among Firsthand delivery formats, but the contents differ and the site makes no compatibility claim.

The comparison table below sets MOKA beside three site entries. The sources read for MOKA do not discuss those methods, so relationship cells say Not stated where that is the case. The RoboPoint entry notes that the RoboPoint paper lists MOKA among methods that still rely on external models to detect task-relevant objects, and the MOKA README is consistent with that, since it adapts its perception from Grounded-SAM, GroundingDINO and SAM. The table compares what each entry states and ranks no methods.

Per the sources read, MOKA prompts a pre-trained VLM with marked RGB images and a free-form instruction, adds a perception pipeline, and can optionally use robot experience for in-context learning and policy distillation. Whether human egocentric video would help a mark-based prompting pipeline is Not stated in the sources read.

Firsthand states human egocentric capture with stereo-derived depth, 21-joint hand pose and one task instruction per episode, and no keypoint labels, mark annotations, affordance labels, masks or synthetic data. No Firsthand data is claimed to have trained or been used by MOKA.

When a program needs real human egocentric demonstrations that public or synthetic data does not cover. Firsthand does not build or train models. Confirm the fields on a Spec Pilot, a small fast first batch; the median from signed spec to first validated batch is 48 hours where coverage is deep, three to four weeks where thin.

What is MOKA?

A published method (arXiv 2403.03174, RSS 2024) that prompts a vision-language model with images annotated with marks to predict point-based keypoint affordances for manipulation tasks given as free-form language. It is third-party research.

What does mark-based visual prompting mean?

Per the abstract and project page, marks such as candidate points, grids and captions are drawn on images so affordance reasoning becomes a series of visual question-answering problems the VLM can solve zero-shot or few-shot.

What does MOKA output for the robot?

Per the project page, a point-based affordance representation made of keypoints and waypoints, produced by the VLM at the low level after it decomposes the task into subtasks at the high level.

What does the official code include?

The official repository is the RSS 2024 implementation under an MIT License. Its README says perception is adapted from Grounded-SAM, GroundingDINO and SAM, the robot platform is built on DROID, and an OpenAI API key is needed.

Does Firsthand use MOKA, and can I request a custom dataset?

Firsthand is not affiliated with the MOKA authors and claims no use, training or compatibility. Keypoint labels, mark annotations, affordance labels, masks, synthetic data, robot trajectories and rollouts are not Firsthand deliverables. You can request a custom dataset.