What Is CogACT?

CogACT is introduced in CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation (arXiv 2411.19650) by Qixiu Li, Yaobo Liang and co-authors, with an official project page, a GitHub repository and Hugging Face checkpoints. The abstract says that existing VLAs adapted from pretrained vision-language models show promising generalizability, but that their task success rates are still unsatisfactory. Its answer is a componentized architecture with a specialized action module conditioned on the VLM output, in place of directly repurposing the VLM for action prediction by simple action quantization. The authors study the action module’s design and report that diffusion action transformers improve performance and scale favorably. This page is a definition, not a guide to running the code.

CogACT is third-party research. Firsthand is not affiliated with its authors or with Microsoft, and makes no claim that CogACT or any other named model was trained, fine-tuned or evaluated on Firsthand data. Firsthand captures first-person human video, collected fast, on demand and to a written spec, not robot or teleoperation data, and it does not deliver robot actions. The raw fields the site states are 3D hand pose at 21 keypoints per hand with a per-joint visibility flag, action segments, and 6-DoF IMUs at 200 Hz on one clock. The only language field is one task instruction per episode. Whether human hand-pose data suits a model of this kind is Not stated in the sources read.

The project page describes a vision module, a language module and an action module. The vision module turns raw images into perceptual tokens using two pretrained vision transformers. The language module integrates the visual information with the language instruction and performs what the authors call cognitive reasoning, using a language-model backbone. The action module receives the resulting cognition feature as an input condition and generates a series of actions.

The vision and language modules are built on a pretrained 7B vision-language model and fine-tuned end to end with an action module of up to 300M parameters, according to the project page. The Hugging Face card for the Base checkpoint lists the components as DINOv2 and SigLIP vision backbones, Llama-2 as the language model, and a DiT-Base action model. The GitHub repository releases three sizes, Small, Base and Large.

The abstract contrasts CogACT with earlier work that repurposes a vision-language model for action prediction through simple action quantization. In CogACT the VLM does not emit the action. It produces one cognition feature, and a separate module decodes that feature into actions. The project page explains the choice of diffusion: real-world physical actions are continuous and often multi-modal, so the authors predict them with a diffusion modeling process and a Diffusion Transformer backbone.

The GitHub README adds a speed comparison on one GPU in bfloat16. It reports about 181 ms per inference for the Base model against 307 ms for OpenVLA, with 16 actions generated per inference against 1. Its stated reason is that a single cognition token generates a whole action sequence, while an OpenVLA-style model generates 7 tokens to represent a 7-dimensional action. These are the repository authors’ own measurements on their own hardware.

The project page says the primary training data is a subset of the Open X-Embodiment dataset, and that the model was pretrained on 0.4M trajectories from it. It adds that the model adapts to a new robot and environment with a few hundred trajectories. For one real-world robot, the authors say they collected a dataset of 391 demonstrations and fine-tuned the compared models on it.

The abstract reports that CogACT exceeds the average success rates of OpenVLA, which has a similar 7B size, by over 35% in simulated evaluation and 55% in real robot experiments, and outperforms the 55B RT-2-X model by 18% absolute success rate in simulation. Those are the authors’ own results on their own evaluations. The number of robot embodiments is given differently on the abstract and on the project page, so this page does not state it.

The Hugging Face card says CogACT takes a language instruction and a single-view RGB image and predicts the next 16 normalized robot actions, each a 7-DoF end-effector delta of the form x, y, z, roll, pitch, yaw, gripper. The actions are then unnormalized using dataset statistics and can optionally be combined by the Adaptive Action Ensemble, an inference strategy the project page says aggregates predicted actions according to their similarity.

For custom data, the README says to collect real-world operation data, for example with teleoperation, convert it to RLDS with actions as end-effector deltas, and fine-tune the released checkpoints. It recommends full fine-tuning over LoRA and says around 30 epochs already yields good results. Those are the repository’s own recommendations.

A model with this input and output needs episodes that pair an image stream, an instruction and a time-aligned action signal. Firsthand does not deliver robot action trajectories, and the site states no mapping from its fields to an end-effector delta format. The last table sets those needs against the raw fields the site states, and any such mapping stays a modeling decision in your pipeline.

If your program needs different fields, signals or environments, collection is custom, fast and on demand, scoped by a written spec. Settle the fields, the delivery format and the wording of the single task instruction per episode on a Spec Pilot, a small first batch, then request a custom dataset.

What does CogACT stand for?

The paper title describes synergizing cognition and action in robotic manipulation. The sources read do not spell the acronym out further.

How is CogACT different from a token-based VLA?

Per the abstract, a token-based VLA repurposes the VLM for action prediction through simple action quantization. CogACT instead conditions a separate diffusion transformer action module on the VLM output.

Is CogACT open source?

The GitHub repository and the Hugging Face card both state that the code and model weights are licensed under the MIT license.

Does CogACT use action chunking?

The released model predicts 16 actions per inference, per the Hugging Face card, and the README notes that an action chunking strategy can output several actions without ensembling.

Does Firsthand deliver data in CogACT format?

No. Firsthand delivers raw fields from first-person human capture, such as 3D hand pose, action segments and IMU, and does not ship robot actions. Mapping them to end-effector deltas is Not stated and stays in your pipeline.

What language data does Firsthand provide?

One task instruction per episode, paired with the video and sensor streams. The site states no narration or per-step language labels.