Glossary
What Is Point Policy (Key Points for Robot Manipulation)?
Last reviewed: November 2026
Short answer
Point Policy is the method from the paper “Point Policy: Unifying Observations and Actions with Key Points for Robot Manipulation” (arXiv 2502.20391). It learns robot policies exclusively from offline human demonstration videos, with no teleoperation data, by turning human hand poses into robot poses and describing objects with key points. Firsthand is not affiliated.
01
What is Point Policy, and what problem does it address?
Point Policy is named in Point Policy: Unifying Observations and Actions with Key Points for Robot Manipulation (arXiv 2502.20391) by Siddhant Haldar and Lerrel Pinto of New York University; the project page lists CoRL 2025. The authors say each robot data point must be physically executed in the real world, which makes collection costly, so they look for alternative data sources and frameworks that can learn from them. See also the site’s cross-embodiment learning entry.
02
Where does Firsthand fit next to Point Policy?
Firsthand is not affiliated with the Point Policy authors. No named model, Point Policy included, was trained on Firsthand data, and Firsthand does not claim to use any hardware named in the paper: its robot experiments use a Franka Research 3 arm with a Franka Hand gripper, the Deoxys controller and two third-person Intel RealSense D435 cameras. Firsthand captures real human egocentric video, custom collection, fast and on demand, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license, with one task instruction per episode. Key points, point tracks, object point annotations, triangulated 3D points, retargeted robot poses, gripper states, robot trajectories, action labels, reward labels, co-training mixtures, pretrained representations, embeddings, teleoperation data, paired human-robot video, synthetic data and other language annotations are not Firsthand deliverables.
03
How does Point Policy turn human video into robot actions?
Per the paper, Point Policy works in three steps. First, it computes motion tracks of key points on the human hand and on task objects from human videos, using a hand pose detector (MediaPipe) and one annotated frame per task; points from two camera views are triangulated into 3D. Second, a transformer policy (BAKU) is trained to predict future robot points from those key points. Third, at inference the predicted 3D robot points are converted back into a 6-DoF end-effector pose with rigid-body geometry constraints, and the gripper state is predicted as an extra token.
The same representation describes what the policy sees and what it outputs, which the authors call morphology-agnostic. The robot position is the midpoint between the index fingertip and thumb tip, and the gripper counts as closed when the two tips are under 7 cm apart. Actions run on the robot with end-effector position control at 6 Hz.
04
What data and setup does the paper use?
The authors collect 190 human demonstrations across 8 real-world tasks, recorded at 20 Hz and subsampled to about 6 Hz. Object points start from one user-annotated frame; DIFT transfers the annotated points to the first frame of every other demonstration and Co-Tracker follows them through each video.
The paper lists two assumptions: the human hand pose in the first frame is known for each task, and the scene is calibrated, with camera intrinsics, extrinsics and camera-to-robot transforms known. The authors say their datasets, training code and evaluation code are public; the GitHub repository is MIT licensed for the code, and the licence of the data is Not stated in the sources read.
05
What did the authors report, and what are the limits?
The abstract reports an overall 75% absolute improvement over prior works on 8 real-world tasks when evaluated in identical settings as training. The project page reports an average success rate of 88% across those tasks and 74% on novel object instances across 6 tasks, and says the method is robust to background clutter on 3 tasks. These are the authors’ figures on their setup; Firsthand has not verified them.
The paper names limitations: reliance on existing vision models, so failures in hand pose detection or point tracking under occlusion hurt performance; point abstractions give up scene context; and all experiments use a fixed third-person camera view, with extension to egocentric views left as future work.
06
How does Point Policy compare with Track2Act, Any-point Trajectory Modeling, Motion Tracks and EgoZero?
The last table sets Point Policy beside four site entries. Per the sources read, Point Policy uses 3D key points as both policy input and output and needs no teleoperated robot data, while the track-based entries work with 2D tracks.
07
What data does a key-point approach need, and what does Firsthand state?
Per the paper, the method needs human demonstration video from two calibrated third-person views, a hand pose detector, one annotated frame per task and a known initial hand pose. Firsthand states real human egocentric capture, depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and one task instruction per episode. That is a note about format, not a compatibility claim: the paper’s experiments use fixed third-person cameras, and the key points, triangulation and robot conversion come from tools the buyer runs.
No off-the-shelf dataset for this is stated here, so this is custom collection: request on-demand collection and confirm it on a Spec Pilot, with a median of 48 hours to a first validated batch where coverage is deep and three to four weeks where it is thin.
08
Point Policy at a glance, as stated in the paper, project page and repository
| Item | What the sources state |
|---|---|
| Paper | Point Policy: Unifying Observations and Actions with Key Points for Robot Manipulation (arXiv 2502.20391) |
| Authors | Siddhant Haldar, Lerrel Pinto (New York University) |
| Venue | CoRL 2025, per the project page |
| Learning signal | Offline human demonstration videos only, no teleoperation data |
| Policy | BAKU transformer over robot and object key points |
| Robot hardware named | Franka Research 3, Franka Hand gripper, Deoxys controller |
| Cameras named | Two third-person Intel RealSense D435 views |
| Code licence | MIT (GitHub repository) |
| Dataset licence | Not stated in the sources read |
| Firsthand relationship | None |
10
Point Policy pipeline, as stated in the paper
| Stage | What the paper states |
|---|---|
| Hand to robot pose | MediaPipe key points on the index finger and thumb from two views, triangulated to 3D; position is the midpoint of the tips; gripper closed under 7 cm |
| Object points | One annotated frame per task, DIFT transfer to other demonstrations, Co-Tracker tracking, two-view triangulation |
| Policy | BAKU transformer, history of 10 point observations, action chunk of 20 (about 3 seconds) |
| Action | Predicted 3D robot points converted to a 6-DoF pose with rigid-body geometry, executed at 6 Hz |
11
Point Policy beside related entries on the site
| Site entry | What the entry states | How Point Policy differs (sources read) |
|---|---|---|
| Track2Act | Predicts 2-D point tracks from web video, converts them to end-effector poses and refines with a residual policy | Point Policy trains on human video only and predicts 3D robot key points as the policy output |
| Any-point Trajectory Modeling | Predicts future 2-D point trajectories from action-free video to guide a visuomotor policy | Point Policy converts its own predicted 3D points directly into robot actions |
| Motion Tracks (MT-π) | Motion tracks are the policy’s action, trained on human and robot demonstrations | The paper uses MT-π as a baseline; Point Policy works in 3D and needs no robot demonstrations |
| EgoZero | Learns robot policies from smart-glasses demonstrations with zero robot data | Point Policy uses calibrated fixed third-person views; the paper lists egocentric views as future work |
12
Frequently asked questions
- What is Point Policy?
- Per the paper (arXiv 2502.20391), Point Policy is a method for learning robot policies exclusively from offline human demonstration videos, without any teleoperation data. It is third-party research.
- What does it mean to use key points as observations and actions?
- The policy takes key points on the robot and on objects as input and predicts future robot key points as output. The authors convert those predicted 3D points back into an end-effector pose with rigid-body geometry, so one representation serves both roles.
- Does Point Policy need robot data?
- The authors present it as learning from human videos without teleoperation data or online robot interaction. They also study co-training with teleoperated robot data as a separate analysis.
- What did the authors report for Point Policy?
- The abstract reports an overall 75% absolute improvement over prior works on 8 real-world tasks. The project page reports an average success rate of 88% in-domain and 74% on novel object instances. Firsthand has not verified these figures.
- Does Point Policy work with egocentric video?
- The paper’s experiments use fixed third-person camera views, and it lists extending Point Policy to egocentric views as future work. This page makes no claim that it works with Firsthand egocentric capture.
- Was Point Policy trained on Firsthand data?
- No. Firsthand is not affiliated with the authors and no named model was trained on Firsthand data. Key points, point tracks, robot trajectories and action labels are not Firsthand deliverables.
- How can I request a custom egocentric dataset for human-to-robot learning?
- Request a custom dataset with your objects, interactions and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep.
Related terms
Sources
- Point Policy: Unifying Observations and Actions with Key Points for Robot Manipulation (arXiv 2502.20391)
- Point Policy (official project page, linked from the paper)
- Point Policy (official GitHub repository)
- BAKU: An Efficient Transformer for Multi-Task Policy Learning (arXiv 2406.07539)
- P3-PO: Prescriptive Point Priors for Visuo-Spatial Generalization of Robot Policies (arXiv 2412.06784)
- Emergent Correspondence from Image Diffusion (DIFT, arXiv 2306.03881)
- CoTracker: It is Better to Track Together (arXiv 2307.07635)
- MediaPipe: A Framework for Building Perception Pipelines (arXiv 1906.08172)
- Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learning (arXiv 2501.06994)