What Is ReKep?

ReKep is introduced in ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation (arXiv 2409.01652) by Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang and Li Fei-Fei, first submitted on 3 September 2024. The project page lists Stanford University and Columbia University. The abstract starts from a question: how to formulate constraints that associate the robot and the environment so they are versatile to diverse tasks, free of manual labeling, and optimizable by off-the-shelf solvers in real time. ReKep is the authors’ answer, a visually grounded constraint representation built on 3D keypoints.

Firsthand is not affiliated with the ReKep authors and makes no claim that any model, ReKep included, was trained on Firsthand data. Firsthand captures human egocentric video, not robot data: custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license. Keypoint labels, constraint annotations, point labels, masks, synthetic data, robot trajectories, teleoperation data and language annotations beyond one task instruction per episode are not Firsthand deliverables.

Per the abstract, a ReKep is expressed as a Python function that maps a set of 3D keypoints in the environment to a numerical cost. A task becomes a sequence of such constraints, one per stage, that specify desired relations between keypoints at different stages and any requirement on the transitions between them. Because a constraint is a function of keypoints, it is relational: it describes how points on the robot, a grasped object and other objects should sit relative to each other, rather than a fixed trajectory.

That is the spatio-temporal part of the title. Space comes from the relations between keypoints; time comes from the ordered stages. The authors say a hierarchical optimization procedure then solves for robot actions, represented as a sequence of end-effector poses in SE(3), inside a perception-action loop at a real-time frequency.

The project page describes the automated procedure. Given an RGB-D observation and a free-form language instruction, DINOv2 proposes keypoint candidates on fine-grained, meaningful regions of the scene. The image overlaid with those keypoints and the instruction go to GPT-4o, which writes a series of ReKep constraints as Python programs. A constrained optimization solver then produces a dense sequence of end-effector actions subject to those constraints.

The page states that the entire pipeline involves no additional training and no task-specific data. This is the keypoint-based VLM planning idea: the language model plans in the space of keypoints and code, and a solver, not a learned policy, produces the motion.

The project page says keypoints are tracked in real time, so the system can replan in closed loop, both within a stage and across stages. In its demonstration an operator perturbs objects and the robot, and the robot reacts; the page gives a teapot-pouring example where the robot restores the teapot to upright and tries the pour again after the cup is moved away.

The abstract lists the behaviors shown on a wheeled single-arm platform and a stationary dual-arm platform: multi-stage, in-the-wild, bimanual and reactive, all without task-specific data or environment models.

The official repository, huangwl18/ReKep, is described as the official demo code for ReKep implemented in OmniGibson. Its README notes that the codebase does not contain the real-world perception pipeline, including keypoint tracking, mask tracking and SDF reconstruction; the demo reads that information from the simulation. It points to the paper appendix for real-robot deployment.

The README lists three known limits: sensitivity to the VLM and prompts used, compounding latency because the pipeline runs sequentially, and task-space planning that can occasionally produce actions that are kinematically hard for a robot to reach. Firsthand reproduces no performance figures from the paper.

The comparison table below sets ReKep beside three site entries. The sources read for ReKep do not discuss those methods, so relationship cells say Not stated where that is the case. The table compares what each entry states, and ranks no methods.

Per the sources read, ReKep needs RGB-D observations, a language instruction, large vision and vision-language models, keypoint tracking and a solver, and it uses no task-specific data or environment models. Whether human egocentric video would help a keypoint-constraint planner is Not stated in the sources read.

Firsthand states human egocentric capture with stereo-derived depth, 21-joint hand pose and one task instruction per episode, and no keypoint labels, constraint annotations, masks or synthetic data. No Firsthand data is claimed to have trained or been used by ReKep.

When a program needs real human egocentric demonstrations that public or synthetic data does not cover. Firsthand does not build or train models. Confirm the fields on a Spec Pilot, a small fast first batch; the median from signed spec to first validated batch is 48 hours where coverage is deep, three to four weeks where thin.

What is ReKep?

A published method (arXiv 2409.01652) that represents a manipulation task as Relational Keypoint Constraints, Python functions mapping 3D keypoints to a numerical cost, which a solver optimizes into end-effector poses. It is third-party research.

How does ReKep use a vision-language model?

Per the project page, DINOv2 proposes keypoint candidates, then GPT-4o reads the keypoint-overlaid image and the instruction and writes the constraints as Python programs. A constrained optimization solver turns them into actions.

Does ReKep need task-specific training data?

The abstract and project page say the system performs tasks without task-specific data or environment models, and that the pipeline involves no additional training.

What does the official code include?

The official repository is demo code implemented in OmniGibson. Its README says it does not contain the real-world keypoint tracking, mask tracking and SDF reconstruction pipeline.

Does Firsthand use ReKep, and can I request a custom dataset?

Firsthand is not affiliated with the ReKep authors and claims no use, training or compatibility. Keypoint labels, constraint annotations, masks, synthetic data, robot trajectories and rollouts are not Firsthand deliverables. You can request a custom dataset.