What Are Heterogeneous Pre-trained Transformers (HPT)?

HPT is introduced in Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformers (arXiv 2409.20537) by Lirui Wang, Xinlei Chen, Jialiang Zhao and Kaiming He. It appeared at NeurIPS 2024, and the project page lists it as a spotlight. The abstract names heterogeneity as one of the roadblocks for training generalist robotic models: previous methods often collect data for one embodiment and one task, which the authors call expensive and prone to overfitting. HPT is their answer, a trunk that is pre-trained across embodiments and tasks and then shared. The official sources are the paper, a project page, a GitHub repository and a Hugging Face checkpoint. This page is a definition, not a guide to running the code.

HPT is third-party research. Firsthand is not affiliated with its authors, MIT CSAIL or FAIR, and makes no claim that HPT or any other named model was trained, fine-tuned or evaluated on Firsthand data. Firsthand captures first-person human video, collected fast, on demand and to a written spec. It is not robot or teleoperation data, and Firsthand does not deliver robot proprioceptive state or robot actions. The raw fields the site states are head and wrist camera video, depth derived from a calibrated stereo rig, 3D hand pose at 21 keypoints per hand with a per-joint visibility flag, action segments, and 6-DoF IMUs at 200 Hz on one clock. The only language field is one task instruction per episode. Whether those fields suit an HPT-style model is Not stated in the sources read.

According to the abstract, HPT aligns the specific proprioception and vision inputs from distinct embodiments to a short sequence of tokens, then processes those tokens to map to control for different tasks. The project page describes the same idea as mapping embodiments, each with its own proprioception and vision sensors, onto a shared latent space with embodiment-specific tokenizers. It says HPT is modularized into stems, a trunk and heads, and that the goal is to pre-train a large, shareable transformer.

Only the GitHub README says what the outer parts are made of: its config lets you change the perception stem networks and the action head networks, and its tagline is to put a scalable transformer in the middle of your policy rather than train from scratch. The token count per stem, the exact stem and head designs and the trunk’s depth are Not stated in the sources read. The abstract says only “a short sequence”.

The abstract says HPT leverages recent large-scale multi-embodiment real-world robotic datasets, plus simulation, deployed robots and human video datasets, and that the authors studied scaling behaviors of training objectives to the extent of 52 datasets. The project page says the training mixture contains over 50 datasets and 200k trajectories from real robot teleops, human videos, simulation and deployed robots. Which datasets those are, and how much each source contributes, is Not stated in the sources read.

The README says pre-training uses a mixture of embodiment datasets with a supervised learning objective, and offers checkpoints because pre-training takes time. It lists five: HPT-XLarge at 226.8M parameters, HPT-Large at 50.5M, HPT-Base at 12.6M, HPT-Small at 3.1M and HPT-Base (With Language) at 50.6M. The abstract reports that HPTs outperform several baselines and improve fine-tuned policy performance by over 20% on unseen tasks in multiple simulator benchmarks and real-world settings. That is the authors’ own result. The repository lists the MIT License.

The site defines a vision-language-action model as one that takes a visual scene and a language instruction and outputs an action. The HPT abstract stresses a different pair of inputs: proprioception and vision from each embodiment, aligned to tokens for a shared trunk. It does not mention a language instruction, and the README lists one checkpoint labelled “With Language”, so how HPT handles language is Not stated in the sources read. This page does not classify HPT as a VLA model, because the sources read do not.

The first comparison table sets out only what the sources state for HPT beside what the site’s own VLA entry says.

Two points in the sources bear on data. First, human video is named in the pre-training mixture next to real robot teleop, simulation and deployed-robot data. How human video is used, and whether it passes through a stem, is Not stated in the sources read. Second, HPT takes proprioception as an input, and a first-person human recording does not contain it in robot form.

Firsthand does not deliver robot proprioceptive state, and the site states no mapping from its raw fields, such as 3D hand pose or IMU, to an HPT stem. That mapping belongs in your own pipeline. The last table sets what an HPT-style episode needs against the fields the site states.

The README says training on your own data needs a convert_dataset function that packs the dataset, and that the perception stem and action head configs can be changed. What you need depends on those choices, so settle them first. Firsthand collection is custom, fast and on demand, scoped by a written spec. Agree the fields, the delivery format, the depth encoding and the wording of the single task instruction per episode on a Spec Pilot, a small first batch, then request a custom dataset.

The cross-embodiment learning entry explains how human video sits next to robot data in a pretraining mix, and links onward to mid-training and the embodiment gap.

What does HPT stand for?

Heterogeneous Pre-trained Transformers. The paper is titled Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformers, and it appeared at NeurIPS 2024.

What are the stems, trunk and heads?

Per the project page, HPT is modularized into stems, a trunk and heads. The goal is to pre-train a large, shareable transformer trunk, and the abstract says the heads map its output to control for different tasks. Detailed designs are Not stated in the sources read.

Is HPT open source?

The GitHub repository lists the MIT License, provides training code and pre-trained checkpoints, and the hpt-base checkpoint is on Hugging Face. The README also says a cleaner implementation exists in Hugging Face.

How many datasets was HPT trained on?

The abstract says the scaling experiments went to the extent of 52 datasets. The project page says the training mixture contains over 50 datasets and 200k trajectories from real robot teleops, human videos, simulation and deployed robots.

Does Firsthand deliver data in HPT format?

No. Firsthand delivers raw fields from first-person human capture, such as calibrated stereo-derived depth, 3D hand pose, action segments and IMU, and does not ship robot proprioception or actions. Mapping them to an HPT dataset is Not stated and stays in your pipeline.

What language data does Firsthand provide?

One task instruction per episode, paired with the video and sensor streams. The site states no narration or per-step language labels.