Glossary
What Is Body Pose Estimation?
Last reviewed: October 2026
Short answer
Body pose estimation recovers the 3D positions of body joints — shoulders, elbows, hips, knees and the rest of the skeleton — across a clip, as a per-frame set of joint coordinates. Alongside hand pose, it captures whole-body motion rather than just what the hands are doing at a given moment.
01
Why it matters
Humanoid and other embodied-AI policies need more than hand geometry: a task like carrying, reaching or stepping around an obstacle depends on torso orientation, stance and arm extension together. Body pose gives a policy the whole-body context that hand pose alone leaves out, which matters most for humanoid and whole-body manipulation training rather than fixed-base arm tasks.
02
How Firsthand handles it
Firsthand captures 3D body pose synchronized with 3D hand pose, depth and RGB under two milliseconds, and packs both into the observation.state vector in the LeRobot v2.1 export — hand joints flattened first, then body joints, in a fixed ordering documented in the format guide.
03
Body pose vs. hand pose
| What is estimated | Typical use | |
|---|---|---|
| Body pose | Shoulder, elbow, hip, knee and other body joints, per frame | Whole-body motion: stance, reach, carrying, stepping around obstacles |
| Hand pose | Hand and finger joints (handJoints[2][21][3]), per frame | Fine manipulation: grasp shape and contact at the object |
05
Body pose fields in a Firsthand export
| Field | Format | Notes |
|---|---|---|
| Body joints | Per-frame 3D joint coordinates, metric units | Captured in the head-camera frame, aligned to depth and RGB |
| observation.state (LeRobot) | 2×21×3 hand joints flattened, then body joints | Hands come first in the vector; keep the ordering when you slice it |
| Synchronization | Hardware-aligned under 2 ms | Shared with hand pose, depth and RGB so no stream is a separately estimated signal |
06
Frequently asked questions
- What is body pose estimation?
- Body pose estimation recovers the 3D positions of body joints — shoulders, elbows, hips, knees and the rest of the skeleton — across a clip, as a per-frame set of joint coordinates.
- How is body pose different from hand pose estimation?
- Hand pose covers hand and finger joints for fine manipulation detail like grasp shape. Body pose covers the larger skeleton — torso, arms and legs — for whole-body motion like stance, reach and stepping. Firsthand captures both together, synchronized.
- What does body pose data look like in a robotics dataset?
- Per-frame 3D joint coordinates in metric units, aligned to depth and RGB. In a LeRobot export, body joints are packed into observation.state after the flattened hand joints, in a fixed ordering.
- How is body pose captured or labeled in egocentric data?
- Firsthand captures body pose live on the rig, synchronized with hand pose, depth and RGB under two milliseconds — so it is a captured signal rather than one reconstructed after the fact from RGB alone.
- Why does body pose matter for humanoid and embodied AI training?
- Humanoid and whole-body manipulation tasks depend on torso orientation, stance and arm extension together, not just hand geometry. Body pose gives a policy that whole-body context, which a fixed-base arm task may not need as much.
- What are the limits of body pose from a first-person view?
- A head-mounted egocentric camera sees the world from the wearer’s own viewpoint, so parts of the wearer’s own body can fall outside the frame or be self-occluded depending on the pose, which is a different capture constraint than a third-person, fixed external camera.
- Is body pose captured separately from hand pose?
- No. Firsthand captures both as part of the same synchronized stream set, aligned under two milliseconds, and packs both into the same observation.state vector in the LeRobot export rather than delivering them as separate, unaligned signals.