What Are Motion Tracks (MT-π) for Human-Robot Transfer?

Motion tracks are named in Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learning (arXiv 2501.06994) by Juntao Ren, Priya Sundaresan, Dorsa Sadigh, Sanjiban Choudhury and Jeannette Bohg of Cornell University and Stanford University. The authors say human videos are a scalable alternative to labor-intensive teleoperated robot data but lack robot action labels, so they represent actions as short-horizon 2D trajectories on an image. See also the site’s imitation learning and cross-embodiment learning entries.

Firsthand is not affiliated with the MT-π authors. No named model, MT-π included, was trained on Firsthand data, and Firsthand does not claim to use any hardware named in the paper. Firsthand captures real human egocentric video, custom collection, fast and on demand, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license, with one task instruction per episode. Motion tracks, keypoint tracks, grasp labels, retargeted actions, robot trajectories, action labels, reward labels, co-training mixtures, pretrained representations, embeddings, teleoperation data, paired human-robot video, synthetic data and other language annotations are not Firsthand deliverables.

Per the paper, a motion track is a short-horizon 2D trajectory on an image that captures the predicted direction of motion of a human hand or a robot end-effector. MT-π, the Motion Track Policy, takes image observations and outputs motion tracks as actions, so human video can supervise the same action space as robot demonstrations. The project page describes a Keypoint Retargeting Network that maps robot keypoints toward the human hand structure and a Motion Prediction Network, based on Diffusion Policy, that predicts future keypoint tracks and grasp states.

Tracks are not directly executable. At test time the authors predict them from two third-person camera views and use stereo triangulation with known extrinsics to recover 3D keypoints and a 6DoF trajectory.

For robot demonstrations the authors assume at least one calibrated camera and robot proprioceptive states, projecting the end-effector into the image as keypoints. For hands, an off-the-shelf detector (HaMeR) gives 21 keypoints, of which five are selected to roughly match the gripper, and a proximity heuristic between fingertips and the object mask infers grasps.

The abstract reports an average success rate of 86.5% across 4 real-world tasks, 40% above baselines that use neither human data nor the motion-track action space. On the project page, four table-top tasks gave every method 25 teleoperated robot demonstrations and MT-π an extra 10 minutes of human demonstrations, compared against Diffusion Policy and ACT. The authors name missed grasps and disagreement between per-view track predictions as failure modes. These are the authors’ figures on their setup; Firsthand has not verified them.

The table sets MT-π beside five site entries. Per the sources read, MT-π uses the 2D track as the policy’s own action; Track2Act and Any-point Trajectory Modeling use predicted tracks to plan or guide another policy.

Per the project page, it needs RGB human demonstrations, a hand pose detector, a way to infer grasps, and robot demonstrations with calibrated cameras. Firsthand states real human egocentric capture, 3D hand pose at 21 joints per hand and one task instruction per episode. That is a note about format, not a compatibility claim: the 2D tracks, grasp labels and robot data come from tools the buyer runs.

No off-the-shelf dataset for this is stated here, so this is custom collection: request on-demand collection and confirm it on a Spec Pilot, with a median of 48 hours to a first validated batch where coverage is deep and three to four weeks where it is thin.

What are motion tracks in MT-π?

Per the paper (arXiv 2501.06994), a motion track is a short-horizon 2D trajectory on an image that captures the predicted direction of motion of a human hand or a robot end-effector. MT-π outputs them as its actions. It is third-party research.

Why does human video need a shared action space?

The authors say human videos are a scalable alternative to teleoperated robot data but lack robot action labels. A shared image-space action lets hand motion in human video supervise the same policy as robot demonstrations.

What did the authors report for MT-π?

They report an average success rate of 86.5% across 4 real-world tasks, 40% above baselines that use neither human data nor their action space. Firsthand has not verified these figures.

Was MT-π trained on Firsthand data?

No. Firsthand is not affiliated with the authors and no named model was trained on Firsthand data. Motion tracks, grasp labels, robot trajectories and action labels are not Firsthand deliverables.

How can I request a custom egocentric dataset for human-to-robot learning?

Request a custom dataset with your objects, interactions and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep.