What Is Action Tokenization in VLA Models?

The FAST paper defines action tokenization as a mapping from a sequence of continuous actions to a sequence of discrete tokens from a fixed vocabulary, where the number of tokens can vary between sequences. Autoregressive vision-language-action (VLA) models predict the next token, exactly as a language model does, so the action signal has to be written as tokens first and decoded back into continuous commands afterwards. The FAST authors state that this choice determines how the discrete symbols a model predicts map to continuous robot actions, and that the common per-dimension, per-timestep binning scheme typically performs poorly when learning dexterous skills from high-frequency robot data.

Tokenization starts from continuous, time-aligned action signals, and it is a modeling decision made after capture. Firsthand does not train, ship or benchmark VLA models and does not ship tokenized actions. It captures first-person human video, not robot or teleoperation data, and delivers the raw continuous fields the site already states, hardware-triggered onto one clock, so a buyer’s own pipeline can bin, compress or regress them: 21-keypoint 3D hand pose per hand with a per-joint visibility flag, wrist trajectories, action segments, contact events, 200 Hz IMU and calibrated stereo depth. The only language field is one task instruction per episode. Firsthand is not affiliated with the authors of any paper cited here and makes no claim that any named model was trained or evaluated on its data.

RT-2 states that it represents actions as tokens in the model’s output, treated the same way as language tokens. Its action space is the 6-DoF positional and rotational displacement of the end-effector, the gripper extension and a discrete command to end the episode. The continuous dimensions are discretized uniformly into 256 bins, so one action becomes 8 integers, and 256 tokens in the model are reserved to stand for the bins.

The OpenVLA paper describes a Llama 2 7B backbone that predicts tokenized output actions, which are decoded into continuous actions the robot can execute. The official OpenVLA repository describes its scheme as 256-bin action discretization, and the FAST paper groups RT-2 and OpenVLA together as per-dimension binning. OpenVLA’s own bin ranges are Not stated in the sources read. The FAST authors note that binning each time step can produce hundreds of tokens per action chunk at high control frequency.

FAST (Frequency-space Action Sequence Tokenization) is introduced in FAST: Efficient Action Tokenization for Vision-Language-Action Models. Its premise is that action signals should be compressed before training to reduce correlation between consecutive tokens, borrowing from byte-pair encoding in language models and from the discrete cosine transform (DCT) behind JPEG.

Per the paper, the pipeline normalizes each action dimension (the 1st and 99th quantiles map to the range -1 to 1), applies a DCT to each dimension, scales and rounds the coefficients into a sparse matrix, flattens it with low-frequency components first, and trains a byte-pair encoding tokenizer to compress it losslessly into dense action tokens. Predicted tokens are decoded back into continuous actions. The paper notes the compression is not entirely lossless: a scale parameter sets a trade-off between compression ratio and reconstruction accuracy.

The authors report that 1-second chunks of 14-dimension, 50 Hz shirt-folding data average 700 tokens under binning and 53 under FAST. They also release FAST+, a universal tokenizer trained on 1M real robot action trajectories, and report that FAST combined with π0 matches diffusion VLA performance while cutting training time by up to 5x. These are the authors’ reported results, and the paper notes that the autoregressive π0-FAST is slower at inference than diffusion π0.

π0 adds action outputs that use flow matching to generate continuous action distributions, through a separate action expert. Its paper contrasts this with VLAs that use autoregressive discretization to represent actions like text tokens. Octo tokenizes language and observations as inputs, but its transformer passes readout tokens to output heads, and the paper describes a diffusion head with chunked action prediction through diffusion decoding.

GR00T N1 uses flow matching to learn action generation: a diffusion transformer processes the robot’s state and noised actions, cross-attended with image and text tokens from the VLM backbone, to output denoised motor actions in chunks. The FAST paper adds that other work replaces tokenization with regression heads or diffusion decoding. Whether any of these models also offers a discrete-token variant is Not stated in the sources read.

You need it when the continuous action fields your tokenizer or action head will consume do not exist for your tasks, environments or hand signals. Firsthand collects fast, on demand and to a written spec, billed per validated hour, so the spec is where you settle the fields, the format and the wording of the single task instruction per episode. The site states no mapping from its fields to any specific robot’s action space, so that mapping is Not stated and stays with your pipeline.

What is action tokenization?

It is the mapping from continuous robot actions to discrete tokens from a fixed vocabulary, so an autoregressive VLA can predict actions the way a language model predicts text, then decode them back into continuous commands.

Is action tokenization the same as action chunking?

No. Chunking means predicting a block of future actions at once. FAST tokenizes each action chunk. See the action chunking entry for the policy-design side.

Do all VLA models use discrete action tokens?

No. Per their papers, RT-2 and OpenVLA write actions as discrete tokens, while π0, Octo and GR00T N1 produce continuous actions through flow-matching or diffusion heads.

Why does FAST compress actions instead of binning them?

The FAST authors argue that at high control frequency consecutive binned tokens are highly correlated, which weakens next-token prediction. Compressing the chunk first yields fewer, more informative tokens.

Can the FAST tokenizer be used on my own action data?

The paper describes FAST trained on individual datasets and a released universal tokenizer, FAST+, that it says can be applied as a black-box tokenizer to 1-second action sequences from robot setups. Its use on human hand-pose data is Not stated in the sources read.

Does Firsthand deliver tokenized actions?

No. Firsthand delivers raw continuous fields from first-person human capture, such as 3D hand pose, wrist trajectories, contact events and IMU. Tokenizing them is a step in your own pipeline.

Does Firsthand provide narration or per-step language labels?

No. The only language field the site states is one task instruction per episode, paired to the video and action streams on the same clock.