Glossary
What Is an Action Representation in Robot Learning?
Last reviewed: October 2026
Short answer
An action representation is the way "what was done" gets encoded so a policy can learn from it — it is the format the action half of a training example takes, as opposed to the video or depth that shows what was observed. NVIDIA’s EgoScale paper (arXiv 2602.16710) represents action as a wrist-level pose derived from egocentric video via off-the-shelf SLAM and hand-pose estimation; its GR00T N1 paper (arXiv 2503.14734) instead learns a latent action representation directly from unlabeled video.
01
Why it matters
A robot policy is trained on pairs of state and action: what the scene looked like, and what motion or command produced the next state. How that action half of the pair is represented determines what kind of video can teach a policy anything about motion at all. A representation tied to an explicit, measurable quantity — like a wrist pose or a hand joint position — can be derived from ordinary human video with no robot present, which is what lets EgoScale turn egocentric footage into pretraining signal. A latent representation, as GR00T N1 learns at the base of its data pyramid, instead lets a model infer "what changed" from raw video without ever naming the action explicitly. The choice of representation is what decides whether unlabeled human footage can contribute to a policy’s action learning at all, and how.
02
How Firsthand handles it
Firsthand does not define its own novel action representation or ship a GR00T-style latent action. It captures the measurable quantities a wrist-level or hand-pose action representation is built from — 3D hand pose, wrist trajectories, contact events, body joints, and discrete action segments (each with a start time, end time, verb, and noun) — directly at the time of recording, as part of the same capture session as the egocentric video and depth. That data is collected fast, on demand, and to a written spec, then delivered in RLDS, LeRobot format, or WebDataset with a buyer-owned commercial license, so a buyer assembling their own action representation starts from labels that were measured rather than inferred after the fact.
03
Common action representations in robot pretraining
| Representation | What it encodes | Where it comes from |
|---|---|---|
| Wrist-level pose | The 6-DoF position and orientation of the wrist over time | Measured directly at capture, or derived from egocentric video via SLAM and hand-pose estimation (EgoScale) |
| 3D hand pose | Per-joint 3D position of the fingers and hand | Measured directly at capture, alongside the wrist signal |
| Action segment | A discrete span of the action with a start time, end time, verb, and noun | Annotated directly on the recorded footage |
| Latent action | A learned, compressed representation of what changed between two video frames, with no explicit label named | Inferred by a model directly from unlabeled video (GR00T N1) |
| Robot action (joint / gripper command) | The robot’s own recorded joint angles or gripper command | Recorded directly by the robot during a real trajectory |
05
Which Firsthand fields support which action representation
| Firsthand field (episode schema) | Action representation it supports | Delivery formats |
|---|---|---|
| hand_joints, hand_visible | 3D hand pose | RLDS, LeRobot, WebDataset, HDF5, zarr |
| Wrist IMU / wrist camera streams | Wrist-level pose / trajectory | RLDS, LeRobot, WebDataset, HDF5, zarr |
| action_segments (start_ns, end_ns, verb, noun) | Discrete action segment | RLDS, LeRobot, WebDataset, HDF5, zarr |
| contact_events | Contact-based action signal | RLDS, LeRobot, WebDataset, HDF5, zarr |
06
Frequently asked questions
- What is an action representation in robot learning?
- An action representation is the way "what was done" is encoded so a policy can learn from it — the format the action half of a training example takes, separate from the video or depth that shows what was seen.
- What kinds of action representations are used in published work?
- EgoScale (arXiv 2602.16710) represents action as a wrist-level pose derived from egocentric video via SLAM and hand-pose estimation. GR00T N1 (arXiv 2503.14734) instead learns a latent action representation directly from unlabeled video, without naming an explicit action.
- How does EgoScale represent actions from human video?
- EgoScale derives a wrist-level action representation from egocentric video using off-the-shelf SLAM and hand-pose estimation, so no robot needs to be present at the time of capture.
- How do latent actions in GR00T N1 differ from a wrist-level representation?
- A wrist-level representation is an explicit, measurable quantity — a 6-DoF pose over time. A latent action, as used in GR00T N1, is a learned, compressed representation of what changed between two video frames, inferred by a model without ever naming the action explicitly.
- How is an action representation different from action labels?
- An action label is a specific piece of recorded or derived data — a hand pose, a wrist trajectory, an action segment. An action representation is the broader scheme those labels are organized into so a policy can learn from them, such as a wrist-level pose representation or a latent action representation.
- What does the choice of action representation mean for capturing egocentric data?
- A representation tied to an explicit, measurable quantity — like wrist pose or hand joints — requires calibrated, synchronized capture hardware recording that quantity directly. A latent representation instead requires no special capture-time instrumentation, since it is learned from raw video after the fact.
- Does Firsthand define its own action representation?
- No. Firsthand captures the measurable quantities — 3D hand pose, wrist trajectories, contact events, body joints, and action segments — that a wrist-level or hand-pose action representation is built from, directly at the time of recording, rather than defining a novel representation or a latent-action scheme.
- Can a custom collection be scoped around a specific action representation?
- Yes. A custom collection can be scoped to capture the specific fields — hand pose, wrist signal, action segments, contact events — that a buyer’s chosen action representation is assembled from, to a written spec.
Related terms