Glossary
What Are Latent Actions?
Last reviewed: September 2026
Short answer
Latent actions are a learned, compressed representation of what changed between two video frames — inferred by a model directly from the footage, without ever being told what the underlying action actually was. NVIDIA’s GR00T N1 paper (arXiv 2503.14734) uses latent actions, together with inverse-dynamics pseudo-labeling, to let unlabeled egocentric human video contribute action-relevant training signal even though it carries no explicit robot action label.
01
Why it matters
Most egocentric human video has no robot action label attached to it — there is no joint angle or gripper command recorded alongside the footage, because no robot was involved in capturing it. Latent actions and inverse dynamics are how GR00T N1 addresses that gap at its data pyramid’s wide base layer: rather than requiring an explicit action label on every frame, the model learns a latent action representation from unlabeled video and infers action-relevant signal from the visual change between frames. Without this technique, unlabeled video’s contribution would be limited to general visual pretraining; with it, the same footage also informs the model’s action representation before any robot-specific data enters the pyramid.
02
How Firsthand handles it
Firsthand does not rely on latent actions or inverse-dynamics pseudo-labeling to supply action signal, because its egocentric video, depth, 3D hand pose, and action-segment data are captured and annotated directly at time of recording — collected fast, on demand, and to a written spec, then delivered in RLDS, LeRobot format, or WebDataset with a buyer-owned commercial license. That measured, captured labeling gives a base layer more direct supervisory signal than one relying purely on pseudo-labels inferred after the fact, though the two approaches are not mutually exclusive: a base layer with direct action labels can still be blended with a wider pool of unlabeled video processed through latent-action or inverse-dynamics techniques.
03
Latent actions vs inverse-dynamics pseudo-labels vs measured action labels
| Approach | How the action signal is produced | What it needs at capture time |
|---|---|---|
| Latent actions | A model learns a compressed representation of "what changed between two frames," inferred directly from video | No explicit action label — only the raw video itself |
| Inverse-dynamics pseudo-labels | The action that most plausibly explains an observed state transition is estimated by working backward from the visual outcome | No explicit action label — only the raw video and a state-transition model |
| Measured / captured action labels | Action data (e.g. 3D hand pose, wrist trajectories, action segments) is recorded directly at the time of capture | Calibrated, synchronized capture hardware set up before recording begins |
05
How latent actions fit into the GR00T N1 data pyramid
| Pyramid layer | Typical action-label status | Role of latent actions / inverse dynamics |
|---|---|---|
| Base — web data and egocentric human video | Usually unlabeled for robot action | Lets the base layer contribute action-relevant signal despite lacking an explicit label |
| Middle — synthetic data | Can be labeled directly by the simulator | Not typically needed; the simulator already knows the ground-truth action |
| Top — real robot trajectories | Directly labeled with the robot’s own recorded actions | Not needed; action is measured, not inferred |
06
Frequently asked questions
- What are latent actions?
- Latent actions are a learned, compressed representation of what changed between two video frames, inferred by a model directly from the footage without ever being told what the underlying action actually was.
- How does GR00T N1 use latent actions?
- NVIDIA’s GR00T N1 paper (arXiv 2503.14734) uses latent actions, together with inverse-dynamics pseudo-labeling, to let unlabeled egocentric human video contribute action-relevant training signal at the base layer of its data pyramid, even though that video carries no explicit robot action label.
- What is the difference between a latent action and an inverse-dynamics pseudo-label?
- A latent action is a learned representation of "what changed between two frames" inferred directly from video. Inverse dynamics is a more explicit strategy: estimating the action that most plausibly explains an observed state transition, working backward from the visual outcome to a probable cause. The GR00T N1 paper uses both together.
- How do latent actions differ from measured action labels?
- A measured action label — such as 3D hand pose, wrist trajectories, or an action segment — is recorded directly at the time of capture. A latent action is inferred afterward by a model from the raw video, with no explicit label ever recorded.
- Do latent actions replace the need for measured action labels?
- No. They address a different problem — how to extract some action-relevant signal from video that was never going to carry an explicit label in the first place. A base layer with directly captured labels, like 3D hand pose or action segments, still gives a more direct supervisory signal than one relying purely on inferred latent actions.
- Why do latent actions matter for pretraining on egocentric video?
- Most egocentric human video has no robot action label attached to it, since no robot was involved in capturing it. Latent actions let that unlabeled footage still inform a model’s action representation, rather than being useful only for general visual pretraining.
- Does Firsthand deliver data using latent actions or inverse-dynamics pseudo-labels?
- No. Firsthand captures and annotates action data directly at the time of recording — egocentric video, depth, 3D hand pose, and action segments — rather than inferring action signal from unlabeled footage after the fact.
- Can directly labeled data and latent-action techniques be used together?
- Yes. They are not mutually exclusive: a base layer with direct action labels can be blended with a wider pool of unlabeled video processed through latent-action or inverse-dynamics techniques, which is consistent with how the GR00T N1 pyramid combines layers of differing label density.
Related terms
Sources