Glossary
What Is Motion Retargeting in Robot Learning?
Last reviewed: October 2026
Short answer
Motion retargeting is the mapping of captured human hand and wrist motion onto a robot hand or arm with different kinematics. It converts human joint positions into targets the robot can execute within its own joint limits. In robot learning it is one way to bridge the embodiment gap, so human video can supply action supervision.
01
Why it matters
A human hand is not a robot end-effector. It has different degrees of freedom, a different grasp geometry and a different camera viewpoint, so footage of a person grasping an object does not say which joint angles a robot hand should command. Retargeting is the step that converts the human motion into something the target robot can follow, which is why the site’s guidance names a retargeting or alignment step as needed before human action content maps onto robot joint or end-effector commands.
02
How Firsthand handles it
Firsthand delivers the human side of that pipeline and nothing past it. It captures the measurable quantities a retargeting step reads from — 3D hand pose with per-joint visibility flags, wrist trajectories, contact events and action segments — during recording, from a head-mounted global-shutter camera plus two wrist cameras on one PTP clock, collected fast, on demand and to a written spec. Firsthand does not perform, sell or deliver retargeting, robot joint targets or retargeted data: mapping the human motion onto a particular robot hand is the buyer’s own step. Delivery is RLDS, LeRobot format or WebDataset under a buyer-owned commercial license.
03
What do published studies say about retargeting?
EgoScale (arXiv 2602.16710) gives the most explicit description. It states that it retargets the 21 human hand keypoints into a dexterous robot hand joint space using an optimization-based procedure that enforces joint limits and kinematic constraints, and that it represents arm motion as relative wrist motion between consecutive timesteps, a representation shared across human demonstrations and robot executions.
EgoScale also compares three ways of representing human hand action. It reports that a wrist-only representation performed poorly across all tasks, particularly those needing precise finger articulation and contact timing, and that pretraining with retargeted joint-space hand actions gave the most consistent performance of the three. These are findings from that study and its robot setup.
HumanScale (arXiv 2606.20521) describes egocentric human video as providing only pseudo action labels from human hand retargeting, and states that for its egocentric pretraining set, per-clip end-effector poses and gripper states are estimated from retargeted hand-pose signals as pseudo-action labels, placing the data in the same action space as its robot data.
04
How is retargeting different from cross-embodiment learning?
Retargeting is a mapping applied to motion: human joint positions go in, targets for a different body come out. Cross-embodiment learning is a training strategy: one policy is trained on data from several embodiments, human and robot. They sit at different points in a pipeline and can be combined.
05
How does retargeting relate to action representation and hand pose?
Hand pose estimation is the input. It recovers where the human hand and its joints were, with no robot involved. Retargeting takes that human pose and maps it onto a robot’s own kinematics. The action representation is what the result is encoded as for a policy: in EgoScale, wrist motion plus retargeted hand joint actions.
In Firsthand’s episode schema the human-side inputs are hand_joints, hand_visible, wrist trajectories, contact_events and action_segments, measured at capture and delivered as separate fields on one clock.
06
What does retargeting mean for capturing egocentric data?
If you plan to retarget, the human-side signal sets the ceiling on what the mapping can do, so the capture spec matters before any model is trained. The fields a mapping reads are 21 joints per hand on every frame, a visibility flag for each joint so occluded joints are not mistaken for measured ones, a wrist trajectory, and contact events that mark when a hand took hold of or released an object.
Alignment in time and space comes next. Firsthand streams share one PTP clock domain with int64 nanosecond timestamps, and each rig session is checked against a reprojection-error threshold, rms_reproj_px under 0.28, before its frame-level labels are trusted. The site states no retargeting accuracy figure and no result from retargeting Firsthand data, and this page does not either.
07
What does Firsthand not do?
Firsthand records humans only. It does not perform, sell or deliver retargeting, robot joint targets or retargeted data, and it does not capture robot or teleoperation data. Nothing here claims a model improved because of Firsthand data.
08
How do you get custom egocentric data to retarget?
Custom collection starts with a written skill spec for the tasks, settings and contributor requirements. A Spec Pilot, a small first batch, is the normal starting point. Where contributor coverage is already deep, a signed spec can reach a validated first batch in a median of 48 hours; where contributors need to be recruited first, a realistic timeline runs three to four weeks.
Consent is tracked through a signed consent chain, faces and identifying details outside the task are anonymized, and delivery arrives under a perpetual, buyer-owned commercial license. Capture happens in Firsthand’s contributor settings, not at a customer’s own facility. For cost drivers, see the pricing page.
09
Motion retargeting compared with neighbouring terms
| Term | What it is | How it differs from retargeting |
|---|---|---|
| Embodiment gap | The mismatch between a human body and a robot’s hardware | The problem; retargeting is one way to address it |
| Cross-embodiment learning | Training one policy on data from several embodiments | A training strategy; it can use retargeted actions as inputs |
| Action representation | How actions are encoded so a policy can learn from them | The encoding; retargeted hand actions are one possible content (EgoScale) |
| Hand pose estimation | Recovering 3D positions of the human hand’s joints | The human-side input; no robot is involved |
11
Firsthand fields a buyer’s own retargeting pipeline could read, and why each matters
| Field or stream (episode schema) | What the site says it holds | Why it matters if you retarget |
|---|---|---|
| hand_joints | 3D position of 21 joints per hand on every frame, in metres, in the head-camera frame; float32 [T, 2, 21, 3] | The human keypoints a mapping reads from; EgoScale likewise starts from 21 hand keypoints |
| hand_visible | Whether each hand joint was visible or occluded | Separates seen joints from occluded ones in the input |
| Wrist trajectories | The wrist signal captured alongside hand pose | Source for arm-level targets; EgoScale represents arm motion as relative wrist motion |
| contact_events | When and where a hand made or broke contact with an object | Marks grasp and release timing; EgoScale reports wrist-only supervision struggled where contact timing mattered |
| action_segments | start_ns, end_ns, verb and noun for each sub-action | Says which sub-action each stretch of motion belongs to |
| Episode clock and calibration | All streams on one PTP domain with int64 nanosecond timestamps; sessions checked at rms_reproj_px under 0.28 | Keeps hand, wrist and video aligned in time and space before any mapping |
12
Frequently asked questions
- What is motion retargeting in robot learning?
- Motion retargeting maps captured human hand and wrist motion onto a robot hand or arm with different kinematics, converting human joint positions into targets the robot can execute within its own joint limits. It is one way to bridge the embodiment gap.
- Why does human video need retargeting?
- The body in the footage is not the body that has to execute the action. A human hand differs from a robot end-effector in degrees of freedom, grasp geometry and viewpoint, so a retargeting or alignment step is needed before human action content maps onto robot commands.
- What does EgoScale report about retargeting?
- EgoScale (arXiv 2602.16710) states that it retargets 21 human hand keypoints into a dexterous robot hand joint space with an optimization-based procedure that enforces joint limits and kinematic constraints.
- Is retargeting the same as cross-embodiment learning?
- No. Retargeting is a mapping applied to motion so it fits a different body. Cross-embodiment learning is training one policy on data from several embodiments. A pipeline can use both.
- Does Firsthand deliver retargeted data or robot joint targets?
- No. Firsthand delivers human capture data: 3D hand pose with visibility flags, wrist trajectories, contact events and action segments. It does not perform, sell or deliver retargeting, robot joint targets or retargeted data.
- What should a capture spec include if I plan to retarget?
- The human-side fields a mapping reads: 21-joint hand pose per hand with visibility flags, wrist trajectories, contact events and action segments, all on one clock, scoped in a written skill spec.
Related terms