What Is Seer?

Seer is introduced in arXiv 2412.15109 by Yang Tian, Sizhe Yang, Jia Zeng, Ping Wang, Dahua Lin, Hao Dong and Jiangmiao Pang (v1 19 Dec 2024). The project page and repository cite it as an ICLR 2025 Oral. Read: the arXiv abstract page, project page and repository README; the full paper text was not read.

Third-party research; Firsthand is not affiliated and claims no use, training or compatibility with it. Firsthand captures first-person human video, not robot data, fast, on demand and to a written spec: 21 keypoints per hand with a visibility flag, wrist trajectories and action segments, plus one task instruction per episode.

The abstract splits scalable manipulation-policy learning into two camps: action, meaning behavior cloning from extensive robotic data, and vision, meaning pre-training representations or generative models, also called world models, on large-scale visual datasets. Seer is proposed as an end-to-end paradigm that closes the loop between vision and action.

The abstract defines a Predictive Inverse Dynamics Model (PIDM) as a model that predicts actions using inverse dynamics conditioned on the robot’s forecasted visual states. Rather than mapping an observation straight to an action, it forecasts what the scene should look like next, then infers the action toward that forecast. The authors argue that closing this loop makes the end-to-end PIDM a better scalable action learner.

The project page names three parts: a Multi-Modal Encoder, Conditional Visual Foresight and Inverse Dynamics Prediction. The encoder adds a foresight token, [FRS], and an action token, [INV], both attending to RGB images, language tokens and robot state tokens; [INV] also attends to [FRS]. Encoded [FRS] plus mask tokens reconstructs future RGB images, and encoded [INV] speculates intermediary actions.

The sources read do not call Seer a VLA. Its encoder takes language tokens with RGB images and robot state and outputs actions, so it sits close to the site’s vision-language-action model entry; that grouping is this page’s reading, not the authors’.

The abstract says Seer is initially pre-trained on large-scale robotic datasets, such as DROID, and adapts to real-world scenarios with a little fine-tuning data. The project page adds that performance improves with model size. The README documents quick real-world training with or without pre-training, and a pre-training route. Dataset and model sizes were not read.

The abstract states improvements of 13% on LIBERO-LONG, 21% on CALVIN ABC-D and 43% in real-world tasks, and a new state of the art on CALVIN ABC-D with an average length of 4.28. The README lists CALVIN ABC-D checkpoints: Seer at 3.98 and Seer Large at 4.30 average length. These are the authors’ reported figures, not reproduced here.

The repository states the Apache 2.0 licence for all assets and code unless specified otherwise. Its README checks off real-world experiment code, CALVIN ABC-D code, Seer-Large evaluation and training code, and LIBERO-LONG code. One item, simpleseer, a quick scratch training and deployment code, is unchecked.

The site’s world model entry describes predicting how a scene changes in response to an action. Its inverse dynamics model entry describes working backward from an observed state change to the action that caused it, often to pseudo-label video. Seer joins the two directions in one trained network: it forecasts a future visual state, then infers actions conditioned on that forecast.

Compute, model sizes, dataset sizes and the baselines behind each percentage were not read. The sources do not say whether first-person human video would suit Seer; its pre-training is described on robot datasets.

Seer’s stated pre-training is on robot data, so human video is not its stated input. Firsthand captures human first-person video, not robot data: hand pose, wrist trajectories, action segments and one task instruction per episode; robot state and actions are Not stated. Write tasks, environments and signals into a spec and confirm on a Spec Pilot. Median time to first validated batch: 48 hours where coverage is deep, three to four weeks where thin. Formats: RLDS, LeRobot v2.1, WebDataset, HDF5; matching Seer’s expected fields is the buyer’s to check.

What is Seer?

An end-to-end predictive inverse dynamics model for robot manipulation (arXiv 2412.15109): it forecasts future visual states, then infers the actions toward them.

What is a predictive inverse dynamics model?

A model that predicts actions using inverse dynamics conditioned on the robot’s forecasted visual states, closing the loop between vision and action.

What does Seer claim on benchmarks?

The abstract states improvements of 13% on LIBERO-LONG, 21% on CALVIN ABC-D and 43% in real-world tasks, and an average length of 4.28 on CALVIN ABC-D. These are the authors’ figures, not reproduced here.

Is Seer open source?

The official repository states the Apache 2.0 licence for assets and code unless specified otherwise, and lists released checkpoints.

Does Firsthand use Seer, and can I request a custom dataset?

Firsthand is not affiliated and claims no use, training or compatibility. You can request a custom dataset, confirmed first on a Spec Pilot.