Glossary
What Is 6D Object Pose Estimation?
Last reviewed: October 2026
Short answer
6D object pose estimation is recovering an object’s position and orientation relative to a camera from an image: a 3D translation plus a 3D rotation, six degrees of freedom in all. The BOP benchmark defines the pose as a rigid transformation from the object model to the camera. Firsthand’s episode schema lists no object-pose field.
01
What is 6D object pose estimation?
Object pose estimation answers two questions about one rigid object: where is it, and which way is it facing? BOP, the Benchmark for 6D Object Pose Estimation, states its mission as capturing the state of the art in 6DoF object pose estimation and related tasks such as 2D object detection and segmentation, for fields such as robotics and augmented reality. Many methods first detect or segment the object and then estimate its pose from that image region.
02
What related signals does egocentric capture provide?
The signals Firsthand states around an object are 3D hand pose (21 keypoints per hand with a per-joint visibility flag), contact events (when and where a hand made or broke contact), depth from an active-stereo sensor at 848×480/30, and action segments, all on one PTP clock. The episode schema lists no object-pose field. The site’s stereo capture pages say object poses are annotated against the stereo geometry, while its hand-object interaction and bagging pages list object pose as not stated. Treat it as something you scope in the skill spec and confirm on a Spec Pilot, not a default field.
03
How is object pose different from hand pose?
Object pose is one transform for one rigid body: a rotation and a translation. Hand pose describes an articulated hand, so it needs many points. Firsthand’s hand_joints field is 21 keypoints per hand, 3D, in metres in the head-camera frame. Neither says whether the two touch; a contact event adds the moment a hand made or broke contact with an object.
BOP pairs each dataset with 3D object models, so a pose refers to a known shape; a hand has no such model. The second table compares the signals.
04
Why does object pose matter for robot manipulation?
A grasp relates a hand to an object. Hand pose says where the fingers are; an object pose would say where the thing between them is and how it is oriented. The BOP paper notes that capturing real images and annotating them with 6D object poses takes significant human effort.
The site states no result showing that object pose improves a robot policy; robot results are Not stated. Human video carries no robot actions, so using it for a specific robot still means bridging the embodiment gap.
05
How is 6D object pose evaluated?
BOP compares an estimated pose with the ground-truth pose using three error functions, listed in the third table. A pose is correct when its error is below a threshold, recall is the fraction of annotated instances with a correct pose, and average recall averages it over several thresholds.
Firsthand states no object-pose accuracy figure, tolerance or benchmark score, so none is given here.
06
How do you request a custom manipulation dataset?
Name the objects, grasps and environments your policy must handle, and say which signals you need: object pose, hand pose, contact events or masks. If object pose matters, write it into the skill spec, including the format and reference frame, which the site does not state, and confirm it on a Spec Pilot, a small fast first batch. This is on-demand custom collection, not a catalog download. Delivery is RLDS, LeRobot, WebDataset or HDF5 / zarr under a perpetual, buyer-owned license.
Paid, consented contributors perform the task, not at a customer’s own facility. Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks. Billing is per validated hour.
07
How BOP defines 6D object pose: components and conventions
| Component | What it is | How BOP records it |
|---|---|---|
| Rotation R | The 3D rotation of the object | 3×3 matrix, saved row-wise as cam_R_m2c |
| Translation t | The 3D translation of the object | 3×1 vector cam_t_m2c, in 1 mm units |
| Pose P = [R|t] | A rigid transformation from object model space to camera space | One entry per annotated instance, with obj_id, in scene_gt.json |
| Object model | The 3D shape the pose refers to | 3D mesh in PLY format, 1 mm units |
| Coordinate systems | The frames the pose is expressed in | Right-handed; camera as in OpenCV, looking along Z |
09
Object pose, hand pose, contacts and depth: what the Firsthand site states
| Signal | What it records | What the Firsthand site states |
|---|---|---|
| 6D object pose | Rotation and translation of a rigid object relative to the camera | No schema field; stereo pages mention it; format, frame and accuracy Not stated |
| 3D hand pose | 21 keypoints per hand with a per-joint visibility flag | hand_joints: float32 [T, 2, 21, 3], metres, head-camera frame |
| Contact events | When and where a hand made or broke contact with an object | contact_events annotation |
| Depth | Per-pixel depth aligned to the camera views and clock | Active-stereo sensor at 848×480/30; depth_mm in step keys |
10
BOP pose-error functions and scores
| Name | What it measures |
|---|---|
| VSD | Visible Surface Discrepancy: considers only the visible object part, so indistinguishable poses count as equivalent |
| MSSD | Maximum Symmetry-Aware Surface Distance: surface deviation in 3D, given pre-identified object symmetries |
| MSPD | Maximum Symmetry-Aware Projection Distance: the perceivable deviation, given object symmetries |
| Average recall | Recall averaged over several thresholds; a dataset score averages the three error functions |
11
Frequently asked questions
- What is 6D object pose estimation?
- Recovering an object’s 3D translation and rotation relative to a camera from an image.
- How is object pose different from hand pose?
- Object pose is one transform for a rigid object; Firsthand’s hand pose is 21 keypoints per hand in 3D.
- Does Firsthand deliver object pose labels?
- Not as a standard schema field. The schema lists hand joints, contact events, depth and action segments. Name object pose in the skill spec and confirm it on a Spec Pilot.
- How does BOP score an estimated pose?
- With three error functions, VSD, MSSD and MSPD, and average recall across thresholds.
- What units and frames does BOP use?
- Translations and 3D models are in 1 mm, in right-handed frames. Units for any Firsthand object pose are Not stated.
- How fast can a custom manipulation dataset be collected?
- Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks.
Related terms