What Is Moto (Latent Motion Token)?

Moto is introduced in Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videos (arXiv 2412.04445) by Yi Chen, Yuying Ge, Weiliang Tang, Yizhuo Li, Yixiao Ge, Mingyu Ding, Ying Shan and Xihui Liu. The abstract says “we introduce Moto”, which names the method, and describes “latent Motion Token sequences” produced by a Latent Motion Tokenizer. The arXiv comments list ICCV 2025. Read: the arXiv abstract page, the official project page and the official GitHub README.

Firsthand is not affiliated with the Moto authors and makes no claim that any model, Moto included, was trained on Firsthand data. Firsthand captures human egocentric video, not robot data: custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license. Latent action or motion-token labels, camera calibration files, multi-view rigs, 3D labels, reward labels, success or failure labels, rollouts, planning annotations and language annotations beyond one task instruction per episode are not Firsthand deliverables.

Per the authors, Moto has three training stages. First, a Latent Motion Tokenizer encodes the key visual motion between video frames into compact latent tokens, in an unsupervised manner, from pure video data.

Second, Moto-GPT is pre-trained with autoregressive next motion token prediction on video and instruction pairs. Third, it is co-fine-tuned on action-labeled trajectories to predict robot actions from learnable action query tokens, while keeping the motion token prediction objective.

The project page says the Latent Motion Tokenizer produces discrete motion tokens from two consecutive video frames. Its decoder is regularized to reconstruct the second frame from the first frame and the tokens, so the tokens capture the motion between the frames.

The authors call the resulting token sequences a bridging “language” of motion, learned from videos without action labels. They argue motion knowledge is closely tied to low-level actions and is hardware-agnostic, which helps transfer to real robot actions.

Moto-GPT learns by predicting the next motion token, the way a language model predicts the next word. After pre-training, the abstract says it can produce semantically interpretable motion tokens, predict plausible motion trajectories and assess trajectory rationality through output likelihood.

The project page adds that the pre-trained model separates successful, failed and random robot trajectories by log-likelihood, which the authors say can provide potential reward signals. Those are model outputs in their experiments, not labels supplied by a data vendor.

The authors use a co-fine-tuning strategy that bridges latent motion token prediction and real robot control. Action-labeled robot trajectories are still needed in the last stage; the pre-training stages need only video.

The project page also says the gap between Moto-GPT and a variant trained without motion tokens widens when little action-labeled data is used for fine-tuning. That is the authors’ result on their own benchmark.

These are the authors’ results on their own benchmarks and robots; Firsthand does not reproduce them and ranks no models. The project page says Moto-GPT is competitive with larger vision-language-action models on SIMPLER, with a GPT-style backbone of 98M parameters, and that it generalizes zero-shot on CALVIN ABC→D using RGB images from a static camera.

They also report real-world pick-place and disassembly tasks, and a gain from additional human video pre-training. This page repeats no success rates, because the figures were not read as stated values in the sources.

The comparison table below sets Moto beside four site entries. In short, Moto belongs to the family of methods that learn a compact representation of change between frames without action labels, and its distinguishing feature is autoregressive next motion token pre-training of a GPT-style model.

The abstract does not use the phrase latent actions; the site’s latent actions entry describes the closely related idea of a learned representation of what changed between two frames.

The authors say Moto can pre-train on human video as well as robot video, and that motion tokens can translate human motions into robot actions. Whether any of that video is egocentric is Not stated in the sources read.

Firsthand states human egocentric capture with stereo-derived depth, 21-joint hand pose and one task instruction per episode. Whether those streams match Moto’s inputs is Not stated, and no Firsthand data is claimed to have trained it.

When a program needs real human egocentric demonstrations that public data does not cover. Firsthand does not build or train models. Confirm the fields on a Spec Pilot, a small fast first batch; the median from signed spec to first validated batch is 48 hours where coverage is deep, three to four weeks where thin.

What is Moto (Latent Motion Token)?

A published method (arXiv 2412.04445) that converts video into latent motion token sequences with a Latent Motion Tokenizer, pre-trains Moto-GPT to predict the next motion token, then co-fine-tunes it on robot actions. It is third-party research.

What is a latent motion token?

A discrete token that the Latent Motion Tokenizer derives from two consecutive video frames. The decoder reconstructs the second frame from the first frame and the tokens, so the tokens capture the motion between them.

Why do the authors call motion tokens a bridging language?

They propose motion-related knowledge that is closely tied to low-level actions and hardware-agnostic, so a sequence of motion tokens learned from video can bridge video data and real robot actions.

Does Moto need action labels?

Not for tokenizer training or pre-training, which use video. The final co-fine-tuning stage uses action-labeled robot trajectories, according to the project page.

Is the Moto code available?

Yes. The official GitHub repository holds training and inference code, and its README says the tokenizer and Moto-GPT weights are released on Hugging Face. The repository lists its license as Other, so read it before use.

Does Firsthand use Moto, and can I request a custom dataset?

Firsthand is not affiliated with the Moto authors and claims no use, training or compatibility. Latent action or motion-token labels, camera calibration files, rollouts and success or failure labels are not Firsthand deliverables. You can request a custom dataset.