What Is Track2Act?

Track2Act is introduced in Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulation (arXiv 2405.01527) by Homanga Bharadhwaj, Roozbeh Mottaghi, Abhinav Gupta and Shubham Tulsiani, from Carnegie Mellon University and FAIR at Meta, first submitted on 2 May 2024 and last revised on 8 August 2024. The abstract starts from a familiar cost: generalization to unseen objects in novel scenes usually relies on a large amount of robot demonstration data. The authors propose using web videos instead to predict plausible interaction plans, and a task-agnostic transformation to turn those plans into real-world robot actions.

Firsthand is not affiliated with the Track2Act authors and makes no claim that any model, Track2Act included, was trained on Firsthand data. Firsthand captures human egocentric video, custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license, with one task instruction per episode. Point tracks, optical flow, 2D or 3D trajectory labels, keypoint labels, robot trajectories, teleoperation data, synthetic data and language annotations beyond that instruction are not Firsthand deliverables.

Per the paper, the model takes an initial image, a goal image that defines the task and a set of points in the initial image, then predicts where each of those points should move over future time-steps. That set of 2-D point trajectories is what the authors call the interaction plan. Because the plan is a set of point motions, it is described as embodiment-agnostic: it says how things in the scene should move, not which robot should move them.

The authors contrast tracks with two alternatives they discuss: hand-object masks, which they say do not capture point correspondences across time, and full RGB frames, which they describe as harder to compute. They also argue that predicting tracks abstracts away appearance and texture, which makes it a simpler target than generating video.

The paper describes three stages. First, a diffusion transformer predicts future point tracks from the initial image, the goal image and the query points. Second, with a depth image of the initial scene, the authors fit a sequence of rigid transforms of the object so that its projected points match the predicted 2-D tracks, then apply those transforms to a gripper pose to obtain an open-loop end-effector plan. Third, a closed-loop residual policy predicts small corrections to that plan at each step.

The reason for the residual stage, per the authors, is that the open-loop plan uses no robot-specific data and so lacks accurate information about contact. Their stated failure modes for the open-loop plan are being unable to grasp the object at the right location and being unable to recover from intermediate failures.

Per the paper, the track predictor is trained on passive web videos that the authors did not collect, drawn from human kitchen video and two large robot datasets, cut into clips of 4 to 5 seconds. Ground-truth tracks are not hand-labeled: the authors run an off-the-shelf point tracker, CoTracker, over the clips. The authors report around 400,000 video clips in total.

The residual policy is trained on a small robot-specific set that the authors report as about 400 teleoperated trajectories across 10 tasks on a Spot robot with a front-facing RealSense camera. The authors describe this as three to four orders of magnitude less robot data than related policy-learning work requires. These are the authors’ figures, and Firsthand has not verified them.

The authors evaluate 20 rollouts at each of four generalization levels across 25 tasks in 5 physical locations. In their Table 2, their full method reports success rates of 70%, 60%, 55% and 40% for mild, standard, combinatorial and type generalization, against 60%, 20%, 0% and 0% for a goal-conditioned behavior cloning baseline trained on the same robot data. The authors also note that tasks remain short-horizon and involve a single object, and that success at the higher generalization levels is still not very high. These are the authors’ figures, measured on their setup; Firsthand has not verified or reproduced them.

The first table sets Track2Act beside three site entries. The Track2Act paper does discuss ATM, which it cites as concurrent work; it does not discuss RT-Trajectory, so that cell says Not stated. Firsthand trajectory data is a measured wrist and hand path, which is a different object from an image-space point track.

Per the paper, the plan stage needs video clips with an initial and a goal frame, and deployment needs an RGBD camera on the robot plus a few robot demonstrations. The paper does not state whether egocentric human capture would work in the video role, so that is Not stated. It does list human video among its training sources.

Firsthand states real human egocentric capture with one task instruction per episode, depth derived from a calibrated stereo rig and no robot demonstrations. That is a note about format, not a claim of compatibility, and any tracks would come from a tracker the buyer runs, not from Firsthand.

When a program needs real human demonstrations that public or synthetic data does not cover, specified field by field. Firsthand does not build or train models. Confirm the fields on a Spec Pilot, a small fast first batch; the median from signed spec to first validated batch is 48 hours where coverage is deep, three to four weeks where thin.

What is Track2Act?

A published framework (arXiv 2405.01527) that learns from web videos to predict point tracks toward a goal image, turns them into robot end-effector poses and refines them with a residual policy. It is third-party research.

What does Track2Act predict?

Per the paper, future 2-D locations of points in an initial image, given the initial image, a goal image and the chosen points.

Where do Track2Act training tracks come from?

The authors generate them by running an off-the-shelf point tracker over web video clips, so no hand labeling of tracks is described.

What results do the authors report?

In their Table 2 the full method reports 70%, 60%, 55% and 40% success across four generalization levels, over 20 rollouts per level. These are the authors’ figures; Firsthand has not verified them.

Does Firsthand use Track2Act, and can I request a custom dataset?

Firsthand is not affiliated with the Track2Act authors and claims no use, training or compatibility. Point tracks, optical flow, trajectory labels, robot trajectories and teleoperation data are not Firsthand deliverables. You can request a custom dataset.