Glossary
What Is GR-1?
Last reviewed: October 2026
Short answer
GR-1 is a GPT-style transformer for language-conditioned robot manipulation from ByteDance Research, introduced in arXiv 2312.13139. It is first pre-trained on video prediction with Ego4D clips, then fine-tuned on robot data to predict actions and future images together. It is unrelated to the humanoid robot of the same name. Firsthand is not part of it.
01
What is GR-1?
GR-1 is introduced in Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation (arXiv 2312.13139), with an official project page and GitHub repository. The abstract calls it a straightforward GPT-style model for multi-task language-conditioned visual robot manipulation. The GitHub README lists ICLR 2024. A humanoid robot also uses the name GR-1; it has nothing to do with this model.
02
Where does Firsthand fit next to GR-1?
GR-1 is third-party research. Firsthand is not affiliated with its authors and claims no use, training or compatibility with it. Firsthand captures first-person human video, not robot data, fast, on demand and to a written spec: 21 keypoints per hand with a visibility flag, wrist trajectories and action segments, plus one task instruction per episode.
03
How is GR-1 built?
Per the paper, GR-1 is a GPT-style causal transformer. A CLIP text encoder handles language and an MAE-pre-trained Vision Transformer handles images. Robot state is the 6D end-effector pose plus a binary gripper status. Learned [OBS] tokens drive future-image prediction, and a learned [ACT] token drives arm and gripper actions. The appendix gives 12 layers, 12 heads and 195M parameters, 46M of them trainable.
04
How is GR-1 pretrained on video and then fine-tuned on robot data, as stated in the sources?
Stage one is language-conditioned video prediction on Ego4D. The paper describes Ego4D as more than 3,500 hours with a natural-language annotation per clip, crops 3-second clips, and collects 800,000 clips holding 8M frames. Stage two fine-tunes on robot data with arm, gripper and video losses together: Smooth-L1 for the arm, binary cross entropy for the gripper and pixel mean squared error for images. The CLIP text encoder and MAE image encoder stay frozen in both stages.
05
Why does video generative pretraining matter for robot manipulation?
The authors argue that a robot trajectory itself contains a video sequence, so forecasting future frames helps the robot anticipate events and pick suitable actions. Their CALVIN ablation (ABCD to D) gives an average length of 3.33 without pre-training or video prediction, 3.82 with video prediction only and 4.21 with both. The abstract reports success rising from 88.9% to 94.9%, and from 53.3% to 85.4% in zero-shot unseen scenes. These are the authors’ own results; Firsthand draws no comparison.
06
How does GR-1 differ from GR-2 and from world models?
GR-1 and GR-2 both pre-train on video, then fine-tune on robot data to predict video and actions together. The GR-2 entry states 38 million pre-training clips against GR-1’s 800,000 Ego4D clips. Whether GR-2 reuses GR-1’s design is Not stated in the sources read. A world model, per the site entry, predicts how a scene changes in response to an action. GR-1 also predicts future images, but outputs robot actions directly, so it overlaps with a world model without being only one. Firsthand ranks none.
07
What do the sources say about robot data and code?
The real-robot setup is a 7-DoF Kinova Gen2. Training data was 1,775 object-transport demonstrations collected with an HTC Vive VR system and 2,856 drawer trajectories. The GitHub repository is Apache-2.0 and documents CALVIN evaluation with released checkpoints. Real-robot data release, training instructions and hours of robot data are Not stated in the sources read.
08
What real-world egocentric video supports video-pretrained policies, and how do you get it custom-collected?
GR-1 pre-trained on existing Ego4D clips. The authors hypothesize that egocentric clips with rich human-object interactions help zero-shot scene generalization, and list comparing manipulation-relevant video with video of any kind as future work. Firsthand captures human first-person video, not robot data. Write tasks, environments and signals into a spec and confirm them on a Spec Pilot. Median time to first validated batch: 48 hours where coverage is deep, three to four weeks where thin. Formats: RLDS, LeRobot v2.1, WebDataset, HDF5; mapping to GR-1’s layout is the buyer’s pipeline.
09
GR-1 inputs, outputs and two-stage training setup, as stated in the official sources
| Item | What the official sources state |
|---|---|
| Policy type | GPT-style causal transformer for multi-task language-conditioned visual robot manipulation |
| Inputs | Language instruction, a sequence of observation images, and a sequence of robot states |
| Outputs | Robot actions and future images, end-to-end |
| Stage 1 | Language-conditioned video prediction on Ego4D; 800,000 clips, 8M frames (paper) |
| Stage 2 | Fine-tuning on robot data with arm, gripper and video losses |
| Frozen parts | CLIP text encoder and MAE image encoder, both stages |
| Size | 195M parameters, 46M trainable (appendix) |
| Firsthand relationship | None |
11
GR-1 beside GR-2 and a world model, as each official source or site entry states
| Item | GR-1 (sources read) | GR-2 (site entry) | World model (site entry) |
|---|---|---|---|
| Learns | Future images and robot actions together | Future video and actions together | How a scene changes in response to an action |
| Pre-training video | Ego4D clips with language annotations; 800,000 clips | Daily human activity video; 38 million clips | Dense frames paired with actions |
| Robot data | CALVIN and a real robot | Robot trajectories; size Not stated | Action, pose or contact event paired with video |
| Action output | Delta XYZ and Euler angles plus a binary gripper (CALVIN) | Cartesian trajectories tracked by Whole-Body Control | Not stated in the site entry |
| Egocentric human video | Ego4D pre-training | Not described | One visual input among several |
12
What GR-1 consumes beside what the Firsthand site states
| Aspect | GR-1 (sources read) | What the Firsthand site states |
|---|---|---|
| Pre-training video | Ego4D clips with natural-language annotations | Head-mounted egocentric video; suitability for GR-1 not claimed |
| Robot state | 6D end-effector pose and binary gripper status | Robot state Not stated; 21 keypoints per hand, visibility flag, wrist trajectories |
| Actions | Arm deltas and a binary gripper, from robot data | Robot actions Not stated; action segments |
| Language | One instruction per task; clip annotations in pre-training | One task instruction per episode |
| File layout | Not stated in the sources read | RLDS, LeRobot v2.1, WebDataset, HDF5 |
| Compatibility | Not claimed | Not claimed |
13
Frequently asked questions
- What is GR-1?
- A GPT-style model for language-conditioned robot manipulation, pre-trained on video prediction and fine-tuned on robot data to predict actions and future images.
- How much video was GR-1 pretrained on?
- The paper states 800,000 three-second Ego4D clips holding 8M frames. Hours of that subset are Not stated in the sources read.
- What is the difference between GR-1 and GR-2?
- Both pre-train on video and then fine-tune on robot data. GR-1 used 800,000 Ego4D clips; the GR-2 entry states 38 million clips. Design differences are Not stated in the sources read.
- Is GR-1 open source?
- The official GitHub repository is Apache-2.0 and documents CALVIN evaluation with released checkpoints. Training instructions are Not stated in the README read.
- Is this the same GR-1 as the humanoid robot?
- No. A humanoid robot shares the name, but this GR-1 is the video-pretrained manipulation model from arXiv 2312.13139.
- Does Firsthand use GR-1, and can I request a custom dataset?
- Firsthand is not affiliated with the GR-1 authors and claims no use, training or compatibility. You can request a custom dataset, confirmed first on a Spec Pilot.