What Is TraceVLA?

TraceVLA is introduced by eight authors from the University of Maryland, Microsoft Research and Capital One. The abstract says large vision-language-action (VLA) models pretrained on robot data still struggle with spatial-temporal dynamics, making them less effective at tasks such as manipulation. The authors’ fix is visual trace prompting: encoding state-action trajectories visually rather than feeding a model multiple past frames, which the paper says are "typically highly visually similar and redundant" and can distract the model. This page draws only on the paper, the official project page, the official GitHub repository and the Hugging Face model card. Every result below is the authors’ own, on their own tasks and robots.

Per the paper, visual trace prompting uses Co-Tracker, an off-the-shelf point-tracking model, to extract dense point trajectories from a window of historical image observations. The starting frame is partitioned into a grid (the paper uses 40×40), and Co-Tracker tracks each grid cell across a time window of 6 frames. The pipeline then identifies "active" trajectories, those whose total movement over the window exceeds a threshold, to separate moving points from a static background, and randomly samples 5 active trajectories. Those sampled trajectories are overlaid onto the robot’s original observation frame as a second image. The model receives both the original image and the trace-overlaid image, separated by a learnable separator token, plus a text prompt telling it a second, trace-marked image follows. During training, the trace image is replaced by the original image with probability α (a dropout mechanism), so the model still works if Co-Tracker fails at test time, for example under poor lighting.

Per the paper, TraceVLA is a 7B-parameter model fine-tuned from OpenVLA, which is itself based on the Prismatic vision-language model trained on Open X-Embodiment. The authors also train a smaller TraceVLA-Phi3, a 4B-parameter model with a Phi-3-Vision backbone, first pretrained on Open X-Embodiment (970K trajectories) using the same recipe as OpenVLA, then fine-tuned with visual trace prompting. Both TraceVLA and TraceVLA-Phi3 are fine-tuned from their respective base models for five additional epochs.

To build the fine-tuning data, the authors applied their visual trace generation pipeline to BridgeData-v2, the Google RT-1 robot dataset, and 120 demonstrations the authors collected from 4 manipulation tasks on their own WidowX-250 robot setup for downstream evaluation. In total this produced about 150,000 robot trajectories annotated with visual trace prompting. The GitHub README states the official training script uses 8 nodes of 8 GPUs each (64 GPUs total), and that a Phi3V-based variant is on a separate code branch. The project page states training memory overhead is under 10GB additional on 8 H100 GPUs, and that per-timestep inference adds about 0.002s for image/text tokens, 0.03s for Co-Tracker tracking and 0.004s for dense point tracking.

The abstract reports evaluations across 137 configurations in the SimplerEnv simulator and 4 tasks on a physical WidowX robot, claiming TraceVLA outperforms OpenVLA by 10% on SimplerEnv and 3.5x on real-robot tasks. The paper’s SimplerEnv results table reports an overall average of 47.7% for TraceVLA versus 40.2% for OpenVLA across three Google Robot tasks (Move Near, Pick Coke Can, Open/Close Drawer) under two settings (visual matching and variant aggregation); TraceVLA-Phi3 scores 44.0% versus OpenVLA-Phi3’s 39.9%. On real-robot tasks, the paper reports TraceVLA completing more successful trials than OpenVLA across 8 WidowX-250 tasks, including 4 tasks the authors describe as unseen for generalization testing; for example, on an unseen pick-place-corn task TraceVLA achieved 8 of 10 successful trials against OpenVLA’s 1 of 10. The GitHub README separately reports SimplerEnv results on the Fractal and Bridge evaluation suites, where TraceVLA-7B scores a 45.8% average on Fractal visual matching against OpenVLA-7B’s 36.5%, and 27.7% average on Bridge against 16.7%.

The paper’s ablations ask whether the gain just comes from fine-tuning on a smaller data subset (the authors say no: fine-tuning OpenVLA without trace prompting only gains 1.1%, and fine-tuning OpenVLA-Phi3 without it loses 0.3%) and whether feeding multiple past frames instead of a trace image would help as much (the authors report it instead reduces performance by 6%, which they attribute to redundant information across similar frames). These are the authors’ own comparisons, and Firsthand draws no conclusion from them.

TraceVLA is a fine-tune of OpenVLA, so the two share an action representation and much of their architecture; per the OpenVLA entry on this site, OpenVLA has 7B parameters, a Prismatic-7B base with a fused SigLIP and DINOv2 encoder, and a Llama 2 backbone, trained on 970k Open X-Embodiment trajectories with no visual trace prompting. TraceVLA’s addition is the trace-overlay input and the 150K-trajectory trace-annotated fine-tuning set.

Magma, per its entry on this site, is a separate model that also tracks motion over time, using Trace-of-Mark (ToM) to predict how marked points, including human hands or robotic arms, move across future video frames, applied during pretraining across robotics images and instructional video. TraceVLA’s visual trace instead overlays historical point trajectories onto the current frame as an additional input at both training and inference time, using Co-Tracker rather than the marks-and-prediction setup Magma describes. The TraceVLA sources read do not mention or compare against Magma, so this comparison draws only on each entry’s own sources, and Firsthand ranks neither.

TraceVLA is a public research release built on robot trajectories, and Firsthand is not affiliated with its authors. Firsthand makes no claim that its data trained, was used with or is compatible with TraceVLA. TraceVLA’s visual traces are derived with Co-Tracker and overlaid as an image input; Firsthand’s wrist trajectories are a different, site-stated field, and the site makes no claim that they are equivalent to, or convertible into, visual traces, point tracks or trace-annotated frames. Firsthand does not state that it delivers any of those.

What the site states is custom egocentric collection of real human demonstrations to a written spec, fast and on demand: head and wrist camera video, 3D hand pose at 21 joints per hand with a per-joint visibility flag, wrist trajectories, action segments and contact events. The only language field is one task instruction per episode. Confirm a spec on a Spec Pilot, a small fast first batch. The median from signed spec to first validated batch is 48 hours where coverage is deep and three to four weeks where it is thin. Billing is per validated hour; delivery is RLDS, LeRobot v2.1, WebDataset or HDF5.

What is TraceVLA?

A 7B-parameter vision-language-action model, fine-tuned from OpenVLA, that overlays Co-Tracker point trajectories on the input image as "visual trace prompting" to improve spatial-temporal awareness, per arXiv 2412.10345.

What is visual trace prompting?

A technique that tracks moving points across a window of past frames with Co-Tracker, overlays the active (moving) trajectories on the current image, and feeds that trace-marked image alongside the original image into the model.

What data was TraceVLA fine-tuned on?

About 150,000 trajectories with visual trace annotations, built from BridgeData-v2, the Google RT-1 dataset, and 120 demonstrations the authors collected on their own WidowX-250 robot.

How does TraceVLA compare with OpenVLA?

The paper reports an overall SimplerEnv average of 47.7% for TraceVLA against 40.2% for OpenVLA, and more successful real-robot trials across 8 WidowX-250 tasks. These are the authors’ own results on their own tasks.

Is TraceVLA open source?

The official GitHub repository is released under the MIT License. Training dataset licences are Not stated in the sources read.

Does Firsthand use TraceVLA, and can I request a custom dataset?

Firsthand is not affiliated with the TraceVLA authors and makes no claim of use, training or compatibility. You can request a custom egocentric dataset: write a spec, confirm it on a Spec Pilot, pay per validated hour.