What Is Proprioception in Robot Learning?

A policy that sees only images has to infer where its own arm and hand are. Robot state tells it directly, which is why published robot models take it as an input next to vision and language. Human egocentric video has no such signal, so a buyer needs to know which human-side signals a dataset carries and which robot-side signals it cannot. A human skeleton is not a robot’s joint state.

Firsthand captures humans, not robots, so it delivers no robot proprioception, joint encoder readings, joint torques or motor commands, and states no robot or teleoperation capture. It captures the human-side signals nearest to a robot’s state: 21-joint 3D hand pose per hand with a visibility flag, a 24-joint body skeleton, wrist trajectories and three 6-DoF IMUs at 200 Hz, all on one PTP clock. The site states no head or camera 6-DoF pose and no trained or shipped policy.

Proprioception is a body sensing its own configuration. In robot learning it means the low-dimensional state a robot reports about itself: joint positions and velocities, gripper or hand state, and the pose of its base or torso. A policy receives it next to image observations and a language instruction.

The papers show what that vector holds on specific robots. GR00T N1 (arXiv 2503.14734) describes its RoboCasa state as “the position and rotation of both the end-effector and the robot base, as well as the gripper’s state,” and, for the GR-1 humanoid, “the joint position and rotation of both arms and hands, along with the waist and neck.”

Proprioception senses the robot itself. Exteroception senses the world around it, through cameras and depth. The action space is neither: it is what the policy outputs, such as target joint angles or relative end-effector motion. In GR00T N1, “the input consists of the robot state, visual observations, and language instruction, and the output is the corresponding motor action.”

The boundary blurs because state and action often share a representation: GR00T N1’s humanoid benchmark describes one “state/action space” of joint positions and rotations. Keep them as separate schema fields.

Cameras show the scene but not always the robot: an arm can leave the frame or a gripper can sit behind an object. Proprioceptive state gives the policy a direct reading of its own configuration.

Published architectures handle it as its own input. GR00T N1 states that “A diffusion transformer (DiT) processes the robot’s proprioceptive state and action,” using an MLP per embodiment to project states and actions of varying dimensions into a shared embedding. EgoVerse (arXiv 2604.07607) describes a policy in which “separate stems handle robot-specific wrist cameras and proprioception.”

No. A head-mounted camera records what the wearer sees, not joint encoders, and a person has no robot joint state to log. EgoScale (arXiv 2602.16710) says so directly: “For robot data, the model conditions on the robot proprioceptive state q_t, while human demonstrations do not provide such signals.” Its workaround is to “replace q_t with a learnable placeholder token,” which keeps one model formulation for human and robot data.

This is the human-video side of the embodiment gap: a robot-state input has to be masked or supplied from another source, such as aligned robot data. The site states no robot-state stream and no result for any placeholder or alignment scheme.

None is an exact substitute, but several describe the wearer’s own configuration and motion. In EgoVerse, 3D hand pose and head pose “serve as proxies for human end-effector motion, enabling cotraining across embodiments.” The signals Firsthand states, hand pose, body pose and wrist IMUs, play a similar role. The site does not say which a policy should treat as state or action; that belongs to the buyer’s retargeting or alignment step.

Two limits apply. Human joints are not robot joints, so a 21-joint hand skeleton has no fixed mapping to a particular gripper or hand. And the site states no head or camera 6-DoF pose, SLAM trajectory, gripper-state stream, joint torque or motor command.

Firsthand captures the human side only. Per frame it records 3D hand pose as 21 joints per hand, in metres in the head-camera frame, with a per-joint visibility flag, plus a 24-joint body skeleton, wrist trajectories and contact events marking grasp and release. Three BMI088 6-DoF IMUs (head and both wrists) sample at 200 Hz, depth is derived from a calibrated stereo rig, and every stream shares one PTP clock.

Delivery is HDF5, RLDS, LeRobot or WebDataset. In the LeRobot export, hand joints are flattened first and body joints follow in observation.state: the closest thing to a state vector on the human side, but human pose, not robot joint state.

What is proprioception in robot learning?

It is a robot’s sense of its own body: joint positions and velocities, gripper or hand state and base or torso pose, given to a policy as observations next to camera images. It differs from exteroception and from the action the policy outputs.

Is proprioception the same as the action space?

No. Proprioceptive state is an input the policy reads; the action space is what it outputs. They can use the same quantities, as in GR00T N1’s humanoid benchmark, but they play different roles.

Why do robot policies take proprioceptive state as an input?

Cameras can miss the robot’s own configuration. GR00T N1 (arXiv 2503.14734) feeds the robot’s proprioceptive state through a diffusion transformer, and EgoVerse (arXiv 2604.07607) gives proprioception its own network stem.

Does egocentric human video contain proprioception?

No. EgoScale (arXiv 2602.16710) states that human demonstrations do not provide it, and uses a learnable placeholder token so one model can train on human and robot data.

Which human-side signals can stand in for robot state?

Hand pose, body pose and wrist inertial data are the nearest analogues. EgoVerse uses hand and head pose as proxies for human end-effector motion. None is an exact substitute, because human joints are not robot joints.

Which of these signals does Firsthand capture?

Firsthand states 3D hand pose at 21 joints per hand with visibility flags, a 24-joint body skeleton, wrist trajectories, contact events and three 6-DoF IMUs at 200 Hz, all on one PTP clock.

Does Firsthand deliver robot proprioception, joint torques or motor commands?

No. Firsthand captures humans, not robots. The site states no robot joint encoder readings, joint torques, motor commands, teleoperation or robot capture.

Does Firsthand state a head pose, a SLAM trajectory or a trained policy?

No. The site makes no head or camera 6-DoF pose or SLAM trajectory claim, and no claim that Firsthand trains or ships a policy.