Glossary
What Are Action Labels in Robot Learning?
Last reviewed: September 2026
Short answer
Action labels are data attached to egocentric video or a robot trajectory that describe what action was taken — a 3D hand pose, a wrist-level pose or velocity, a discrete action segment with a verb and noun, or a robot’s own recorded joint and gripper command. NVIDIA’s EgoScale paper (arXiv 2602.16710) derives its action labels from off-the-shelf SLAM and hand-pose estimation run on egocentric video, giving each clip a wrist-level action representation without a robot ever being present at capture time.
01
Why it matters
A robot policy learns from examples of state paired with action — what the scene looked like, and what was done in response. Raw egocentric video by itself only shows the "what happened" side of that pair; without an action label attached, a model can watch the footage but has no ground truth for what motion or command produced it. That is the gap NVIDIA’s GR00T N1 paper (arXiv 2503.14734) addresses at the base of its data pyramid with latent actions and inverse-dynamics pseudo-labeling, and the gap EgoScale (arXiv 2602.16710) addresses by deriving a wrist-level action label directly from SLAM and hand-pose estimation on egocentric video — EgoScale reports that adding its action-labeled egocentric video to a robot-data baseline improved policy success rate by 54% and that success continued to scale log-linearly as more hours were added, without saturating within the range tested.
02
How Firsthand handles it
Firsthand captures action labels directly at the time of recording rather than inferring them afterward: 3D hand pose, wrist trajectories, contact events, gaze target, body joints, and discrete action segments (each with a start time, end time, verb, and noun) are annotated as part of the same capture session as the egocentric video and depth. That data is collected fast, on demand, and to a written spec, then delivered in RLDS, LeRobot format, or WebDataset with a buyer-owned commercial license — so the action label a buyer receives was measured, not estimated from a state-transition model or a learned latent representation after the fact.
03
Types of action labels used in robot pretraining
| Label type | What it captures | How it is produced |
|---|---|---|
| 3D hand pose / wrist trajectory | The position and orientation of the hand or wrist over time | Measured directly at capture, or estimated from video via SLAM and hand-pose estimation (EgoScale) |
| Action segment | A discrete span of the action with a start time, end time, verb, and noun | Annotated directly on the recorded footage |
| Contact events | When and where a hand or tool made contact with an object | Annotated directly on the recorded footage |
| Latent action | A learned, compressed representation of what changed between two frames | Inferred by a model from unlabeled video, with no explicit label recorded |
| Inverse-dynamics pseudo-label | The action estimated to have caused an observed state transition | Inferred by working backward from the visual outcome, with no explicit label recorded |
| Robot action (joint / gripper command) | The robot’s own recorded joint angles or gripper command | Recorded directly by the robot during a real trajectory |
05
Captured action labels vs pseudo-labels
| Approach | Needs a robot at capture time? | When the label is produced |
|---|---|---|
| Captured (hand pose, action segments, contact events) | No | At time of recording |
| SLAM + hand-pose estimation (EgoScale) | No | At time of recording, derived from the video |
| Latent actions | No | Inferred later, during training |
| Inverse-dynamics pseudo-labels | No | Inferred later, during training |
| Robot action (joint / gripper command) | Yes | At time of recording, by the robot itself |
06
Frequently asked questions
- What are action labels?
- Action labels are data attached to a video or trajectory that describe what action was taken — a 3D hand pose, a wrist trajectory, a discrete action segment with a verb and noun, or a robot’s own recorded joint and gripper command.
- What kinds of action labels do robot models use?
- Published work uses both captured labels (hand pose, wrist trajectories, action segments, contact events, recorded directly at capture) and pseudo-labels inferred after the fact, such as latent actions and inverse-dynamics estimates, which let unlabeled video still contribute action-relevant signal.
- How does EgoScale produce its action labels?
- NVIDIA’s EgoScale paper (arXiv 2602.16710) derives a wrist-level action representation from egocentric video using off-the-shelf SLAM and hand-pose estimation, so no robot needs to be present at capture time.
- What is the difference between a captured action label and a pseudo-label?
- A captured action label — hand pose, an action segment, a contact event — is recorded or derived directly from the footage at, or close to, the time of capture. A pseudo-label, such as a latent action or an inverse-dynamics estimate, is inferred by a model later, during training, with no explicit ground truth ever recorded.
- Why do action labels matter for pretraining?
- A policy needs examples pairing a state with the action taken in response. EgoScale reports that adding action-labeled egocentric video to a robot-data baseline improved policy success rate by 54%, with success continuing to scale log-linearly as more hours were added — evidence that the action label, not just the video, is what makes the added data useful.
- How are action labels delivered?
- Firsthand delivers action labels — hand joints, action segments, contact events, gaze target — alongside the egocentric video and depth in RLDS, LeRobot format, or WebDataset, with the field mapping documented so a buyer can retarget it to their own loader.
- Does every pretraining dataset need action labels?
- No. Unlabeled egocentric video and web-scale image or video data still contribute general visual pretraining, as in the wide base layer of GR00T N1’s data pyramid. Action labels become necessary once the goal is to train or refine a policy’s action representation specifically.
- Does Firsthand use latent actions or inverse-dynamics pseudo-labeling to produce its action labels?
- No. Firsthand captures and annotates action data directly at the time of recording — 3D hand pose, wrist trajectories, contact events, and action segments — rather than inferring an action label from unlabeled footage after the fact.
Related terms