What Is HiRT?

HiRT is introduced in HiRT: Enhancing Robotic Control with Hierarchical Robot Transformers (arXiv 2410.05273; submitted 12 September 2024, last revised 3 February 2025, cs.CV, accepted to CoRL 2024) by Jianke Zhang, Yanjiang Guo, Xiaoyu Chen, Yen-Jen Wang, Yucheng Hu, Chengming Shi and Jianyu Chen. The paper lists Tsinghua University, UC Berkeley and Shanghai Qizhi Institute. The abstract names it: “this paper proposes HiRT, a Hierarchical Robot Transformer framework”. Read: the arXiv abstract page and the full paper text, including the appendices. No official project page or GitHub repository was found or linked in the text read, so code availability is Not stated.

HiRT is third-party research. Firsthand is not part of it or affiliated with its authors, and makes no claim that any model was trained on Firsthand data. What the site states is custom egocentric collection of real human demonstrations, fast and on demand, to a written spec: 3D hand pose at 21 joints per hand, calibrated stereo depth and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset with a buyer-owned commercial license. The only language data the site states is one task instruction per episode. Mapping any delivery to HiRT’s inputs or action format is Not stated.

Per the abstract, VLA models built on pre-trained vision-language model backends generalize well, but backends with billions of parameters bring high computational cost and inference latency. The authors say this limits testing to mostly quasi-static tasks and hurts performance in dynamic tasks that need rapid interaction.

HiRT is their answer: a framework that enables a flexible frequency and performance trade-off.

The paper frames it with dual process theory: System 2 is slow and analytical, System 1 is fast and intuitive. HiRT has two modules. The understanding module is InstructBLIP (7B), a pre-trained vision-language model that turns an image and a language instruction into latent features for long-term scene understanding. The execution module is a compact vision-based action policy that combines recent observations with the latest latent.

The VLM does not output a language command. It outputs continuous features that condition the fast policy, which outputs low-level actions such as the end-effector’s relative position and gripper status.

The paper describes three conditioning strategies: FiLM layers for a convolutional encoder, extra cross-attention layers inside transformer blocks, and prefix tuning of the action head. The appendix gives the fast policy as an EfficientNet-B3 encoder with FiLM layers in simulation, about 35M parameters, and a ViT-B/16 encoder with cross-attention in the real-world setup, 150M parameters.

The authors state that HiRT’s input does not include robot state information, so its generalization comes from visual and language information.

At the first step the VLM encodes the observation and caches a latent. Afterwards the fast policy reads the most recent cached latent at every step while the VLM runs in parallel at its own pace, so the policy runs at close to its own speed. During training the VLM part is fine-tuned with LoRA, the rest is fully fine-tuned, and the objective is imitation learning.

Because the cached latent can be several steps old, the authors randomly pick a past observation as the VLM input during training. An ablation in the appendix reports that this helps when the VLM runs at a longer interval.

The abstract reports that on static tasks the control frequency doubles with comparable success rates, and that on novel real-world dynamic manipulation tasks the success rate rises from 48% to 75%. In Table 1 the authors list 9.8 Hz for HiRT and 4.1 Hz for their Vanilla-VLA reimplementation of RT-2.

In the dynamic tests the target object moved at a constant 1 cm/s, and the authors report 75.0 average success for HiRT against 48.0 for Vanilla-VLA. These are their results on their own setups and baselines; Firsthand does not reproduce them or rank models.

The site’s Hi Robot entry describes a high-level VLM that outputs a language command for a low-level VLA. The site’s RoboDual entry describes a slow VLA generalist guiding a fast diffusion specialist. The site’s FiS-VLA entry describes a fast execution module embedded inside a slow VLM. HiRT sits in the same family: slow VLM, fast low-level policy, run at different frequencies.

The HiRT paper says it grounds the VLM to the robot in an end-to-end manner and passes continuous intermediate representations. Whether it relates directly to Hi Robot, RoboDual or FiS-VLA is Not stated in the sources read.

For the real-world experiments the authors report 2000 trajectories with image observations from wrist and third-view cameras on a Franka Emika Panda. Demonstrations were collected with a remote-operation joystick or with scripted policies. The policy conditions on a free-form language instruction.

On the site, the only language data is one task instruction per episode, and Firsthand captures human egocentric video rather than robot data. Using such video with HiRT is Not stated in the sources read.

The authors write that, due to data constraints, they have not yet applied HiRT to more complex dynamic tasks such as grasping high-speed flying objects or rapidly adjusting object postures. Their dynamic tests used a slowly moving target.

No project page or repository was found in the sources read, so released code, checkpoints and licence are Not stated.

When a program needs real human egocentric demonstrations of tasks, environments or objects that public data does not cover, specified in writing. Firsthand does not build or train models.

Confirm the spec on a Spec Pilot, a small fast first batch. Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks. Billing is per validated hour.

What is HiRT?

A published framework (arXiv 2410.05273) whose name is Hierarchical Robot Transformer. A VLM runs slowly to produce features, and a lightweight policy uses them to act at a high frequency. It is third-party research.

What is a fast low-level policy with a slow VLM?

Per the paper, a two-part design: the slow VLM captures slowly changing information about the scene and instruction, and a small vision-based policy reacts quickly to new observations using the most recent VLM latent.

Why does HiRT matter for robotic control with VLAs?

The authors say large VLA backends are slow, which limits them to quasi-static tasks. HiRT trades some VLM frequency for speed so the controller can handle moving targets.

How much does HiRT improve results?

The authors report doubled control frequency on static tasks with comparable success rates, and a rise from 48% to 75% on novel real-world dynamic tasks. These are their results; Firsthand ranks no models.

Is the HiRT code available?

Not stated in the sources read. No official project page or GitHub repository was found or linked in the paper text, so check the authors’ paper page for updates.

Does Firsthand data train HiRT?

No claim is made. Firsthand is not affiliated with its authors, and the site does not state that any model was trained on Firsthand data. The only language data the site states is one task instruction per episode.