What Is HAMSTER?

HAMSTER is introduced in HAMSTER: Hierarchical Action Models For Open-World Robot Manipulation (arXiv 2502.05485) by Yi Li, Yuquan Deng, Jesse Zhang, Joel Jang, Marius Memmel and colleagues, first submitted on 8 February 2025. The project page lists ICLR 2025 and affiliations at NVIDIA, the University of Washington and the University of Southern California. The abstract starts from a data problem: large foundation models generalize in vision and language, but robotics lacks similar data because robot data is usually gathered through expensive on-robot operation. The authors argue that cheaper off-domain data, such as action-free videos, hand-drawn sketches or simulation data, can be used more effectively by a hierarchical VLA than by a monolithic VLA that fine-tunes a vision-language model to predict actions directly.

Firsthand is not affiliated with the HAMSTER authors and makes no claim that any model, HAMSTER included, was trained on Firsthand data. Firsthand captures human egocentric video, not robot data: custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license. Robot trajectories, teleoperation data, 2D path annotations, action labels in robot action spaces, success or failure labels, rollouts and language annotations beyond one task instruction per episode are not Firsthand deliverables.

Per the abstract, HAMSTER studies a class of hierarchical VLA models. The high-level vision-language model is fine-tuned to produce a coarse 2D path that indicates the desired robot end-effector trajectory, given an RGB image and a task description. That intermediate 2D path then serves as guidance for a low-level, 3D-aware control policy that is capable of precise manipulation.

The authors say this split relieves the high-level model from fine-grained action prediction, while reducing the low-level policy’s burden of complex task-level reasoning. They state that, with this design, the high-level model can transfer across large domain gaps between the off-domain fine-tuning data and real-robot testing, including differences in embodiments, dynamics, visual appearance and task semantics.

The project page describes two training steps. First, a pre-trained vision-language model is fine-tuned on a large, general, off-domain dataset to produce paths and points. Second, a data-efficient policy with 3D input is trained on a small in-domain dataset to produce actions conditioned on images that have the path drawn on them.

At inference, the vision-language model predicts paths for the first observation, conditioned on the text instruction. Those paths are drawn onto all the images the low-level policy sees, and the policy then executes the low-level actions. The page’s example prompt asks for a list of image-space points for the gripper, with open and close gripper tags, using coordinates between 0 and 1. Visual generalization comes from the fine-tuned model, and precise control from the separate low-level policy.

The abstract names action-free videos, hand-drawn sketches and simulation data as the kinds of cheaper off-domain data the hierarchy is meant to exploit. The project page says the off-domain fine-tuning set includes robot simulation data and robot teleoperation data from environments different from the target environment. It also shows a comparison where the simulated RLBench data helps path generation for real-world tasks similar to those in the off-domain data.

The sources read do not state dataset sizes or the number of episodes, so this page states none. The abstract says visual results, code and dataset are provided at the project page; no separate official GitHub link appeared in the text read, so this page links only the paper and the project page.

The abstract says that in real-robot experiments the authors observe an average of 20% improvement in success rate across seven different axes of generalization over OpenVLA, which they describe as a 50% relative gain. The project page reports that HAMSTER also works with different low-level policies: averaged success on new tasks rises from 22.11 to 66.56 with RVT2 and from 16.70 to 72.55 with 3D-DA when each is paired with HAMSTER.

On path generation, the project page gives a human-rank comparison against RT-Trajectory, where lower is better, with HAMSTER ranking better than the zero-shot GPT-4o variants used for RT-Trajectory. These are the authors’ own figures from their own setup; read the paper for how they were measured. This page ranks no methods.

The comparison table below sets HAMSTER beside four site entries, using only the sources read for HAMSTER. The project page compares directly with RT-Trajectory, which uses a pre-trained VLM to generate paths without fine-tuning, and with OpenVLA. The sources read do not discuss Hi Robot or RoboDual, so those rows say Not stated.

In short, a standard VLA maps an image and an instruction to an action, HAMSTER inserts a predicted 2D path between a fine-tuned vision-language model and a low-level policy, and RT-Trajectory conditions a policy on a trajectory sketch. This page ranks no models.

The sources read do not give HAMSTER’s parameter counts, task counts, training compute or the hardware of the real-robot setup, so this page states none of them. Whether a different high-level model or a different kind of off-domain data would change the results is not covered here either.

HAMSTER consumes off-domain data for its high-level model and a small in-domain dataset with path-drawn images for its low-level policy. Firsthand states human egocentric capture with stereo-derived depth, 21-joint hand pose and one task instruction per episode, and no robot trajectories, 2D path annotations or action labels.

Whether those streams suit a HAMSTER-style pipeline is Not stated, and no Firsthand data is claimed to have trained or been used by HAMSTER.

When a program needs real human egocentric demonstrations that public data does not cover. Firsthand does not build or train models. Confirm the fields on a Spec Pilot, a small fast first batch; the median from signed spec to first validated batch is 48 hours where coverage is deep, three to four weeks where thin.

What is HAMSTER?

A published hierarchical vision-language-action model (arXiv 2502.05485). A fine-tuned vision-language model predicts a coarse 2D path of the robot end-effector from an image and a task description, and a low-level 3D-aware policy uses that path as guidance. It is third-party research.

What does hierarchical VLA mean in HAMSTER?

The work is split in two. The high-level model handles task understanding and outputs a 2D path, and the low-level policy handles precise control. The authors say this lets the high-level model transfer across domain gaps without having to predict fine-grained actions.

What results do the HAMSTER authors report?

The abstract reports an average 20% improvement in success rate across seven axes of generalization over OpenVLA, a 50% relative gain. These are the authors’ figures; read the paper for how they were measured.

How is HAMSTER different from RT-Trajectory?

Per the project page, RT-Trajectory uses a pre-trained VLM to generate paths without fine-tuning, while HAMSTER fine-tunes the VLM on off-domain data to produce paths and points. The page reports HAMSTER ranking better in a human evaluation, as the authors’ figure.

Does Firsthand use HAMSTER, and can I request a custom dataset?

Firsthand is not affiliated with the HAMSTER authors and claims no use, training or compatibility. Robot trajectories, 2D path annotations, action labels, rollouts and success or failure labels are not Firsthand deliverables. You can request a custom dataset.