What Is Any-point Trajectory Modeling (ATM)?

ATM is introduced in Any-point Trajectory Modeling for Policy Learning (arXiv 2401.00025) by Chuan Wen, Xingyu Lin, John So, Kai Chen, Qi Dou, Yang Gao and Pieter Abbeel, first submitted on 28 December 2023 and last revised in July 2024. The abstract starts from a bottleneck: more demonstration data often improves policy learning, but collecting it is costly. Videos hold knowledge about behaviors, physics and semantics, yet they have no action labels, so extracting control-specific information is hard. ATM is the authors’ answer: learn from video by predicting where points will move.

Firsthand is not affiliated with the ATM authors and makes no claim that any model, ATM included, was trained on Firsthand data. Firsthand captures human egocentric video, custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license, with one task instruction per episode. Point tracks, optical flow, 2D or 3D trajectory labels, keypoint labels, robot trajectories, teleoperation data, synthetic data and language annotations beyond that instruction are not Firsthand deliverables.

Per the paper, a point track is a sequence of 2-D camera-frame coordinates for one point across a video. The model takes the current image, any set of 2-D query points in that image and a language instruction, then predicts where those points will be over a number of future steps. Because tracks are predicted in the camera frame, the authors state they avoid assuming calibrated, multi-view or depth-equipped cameras, which they say allows future scaling to more diverse video.

The authors argue that points capture motion while leaving out texture and lighting, and that they build in ideas such as object permanence and continuous motion. They contrast this with video prediction models, which model every pixel and which they describe as prone to hallucination and computationally demanding.

Stage one trains a track transformer on an action-free video dataset, where each episode has a language instruction. Training tracks are not hand-labeled: the authors run an off-the-shelf point tracker over the videos to generate them, which they call self-supervised track annotation. Because a static camera makes most random points background, they filter out points that barely move by thresholding the variance of position over time.

Stage two trains a track-guided policy on a small set of action-labeled demonstrations. The policy takes the current observation plus the predicted tracks from the frozen track transformer and outputs actions. The authors report that, with fine-grained subgoals supplied by the tracks, the policy no longer needs the language instruction as its task specification.

Per the paper, track, image and language inputs are encoded into a shared embedding space. Future point positions are masked out, each point becomes one token, the language instruction goes through a pre-trained BERT encoder, and image patches are partly masked, with patch reconstruction as an auxiliary task. A transformer then decodes the track tokens into future trajectories.

The paper and the project page give different sizes for the fixed set of points used at policy time, so this entry repeats neither, and it gives no parameter counts.

The abstract says that across over 130 language-conditioned tasks, evaluated in simulation and the real world, ATM outperforms strong video pre-training baselines by 80% on average, and shows transfer from human videos and from videos of a different robot morphology. The paper attributes the average to the LIBERO benchmark setting with 10 action-labeled and 50 action-free demonstrations per task. These are the authors’ figures, measured on their setups, and Firsthand has not verified them or reproduced them.

The first table sets ATM beside three site entries. The sources read for ATM do not discuss RT-Trajectory or LLARVA, so relationship cells say Not stated where that is the case. Note that Firsthand trajectory data is a measured wrist and hand path, which is a different object from an image-space point track.

Per the paper, stage one needs video with a language instruction per episode and stage two needs a few action-labeled demonstrations. The paper does not specify whether egocentric human video would work in that role, so that is Not stated. The authors do show transfer from human videos.

Firsthand states real human egocentric capture with one task instruction per episode, so the episode and instruction shape is similar. That is a note about format, not a claim of compatibility, and the tracks themselves would come from a tracker the buyer runs, not from Firsthand.

When a program needs real human demonstrations that public or synthetic data does not cover, specified field by field. Firsthand does not build or train models. Confirm the fields on a Spec Pilot, a small fast first batch; the median from signed spec to first validated batch is 48 hours where coverage is deep, three to four weeks where thin.

What is Any-point Trajectory Modeling (ATM)?

A published framework (arXiv 2401.00025) that pre-trains on action-free video to predict future trajectories of arbitrary points in a frame, then uses those tracks to guide a policy learned from few action-labeled demonstrations. It is third-party research.

What does ATM predict?

Per the paper, future 2-D camera-frame positions of query points, given the current image, the points and a language instruction.

Where do ATM training tracks come from?

The authors generate them by running an off-the-shelf point tracker over action-free videos, so no hand labeling of tracks is described.

What results do the authors report?

The abstract reports that across over 130 language-conditioned tasks in simulation and the real world, ATM outperforms strong video pre-training baselines by 80% on average. These are the authors’ figures; Firsthand has not verified them.

Does Firsthand use ATM, and can I request a custom dataset?

Firsthand is not affiliated with the ATM authors and claims no use, training or compatibility. Point tracks, optical flow, trajectory labels, robot trajectories and rollouts are not Firsthand deliverables. You can request a custom dataset.