Egocentric Video with Body Pose Data

Egocentric video with body pose data is first-person task footage delivered together with the 3D positions of the wearer’s body joints over time. On Firsthand’s rig that is a 24-joint skeleton per frame, delivered as body_joints in the annotations group of the episode schema, on the same timeline as the video, the hand joints, depth and IMU.

This page sells the collection: fast, on-demand, custom collection scoped to your skill spec, with body pose delivered beside the video. It is not the definition and it is not annotation of footage you already own. The body pose estimation glossary entry defines the term, and the keypoint and pose annotation page covers labeling. Here the focus is what you receive and how to order it.

The site’s glossary states the case: humanoid and other embodied-AI policies need more than hand geometry, because a task like carrying, reaching or stepping around an obstacle depends on torso orientation, stance and arm extension together. It adds that body pose matters most for humanoid and whole-body manipulation training rather than fixed-base arm tasks.

The humanoid pages say the same thing from the data side: whole-body pose alongside the hands carries the balance and locomotion context a humanoid action depends on, and body pose with IMU adds stance, reach and the transfer between locomotion and manipulation. The site states this dependency but does not quantify how much body pose changes model performance, and this page does not either.

Body pose is delivered as body_joints in the annotations group, listed beside hand_joints, hand_visible, contact_events, instance_masks, objects, action_segments and gaze_target. The site’s format FAQ gives its shape as bodyJoints[24][3]: 24 joints with three coordinates each. The stream list describes it as 3D, root-relative and world-frame, with torso and forearm chains resolved, and the glossary describes the coordinates as metric and captured in the head-camera frame.

The two frame descriptions are both on the site, so which frame a given delivery uses is confirmed in the spec rather than assumed. The body pose field sits next to the hand fields, which are better specified: float32 [T, 2, 21, 3] in metres in the head-camera frame, with hand_visible flagging occluded joints. The first table lists each field and what the site states about it.

They are three different signals. Body pose is the larger skeleton, covering torso, arms and legs, and is used for whole-body motion such as stance, reach and stepping. Hand pose is the 21 joints per hand used for fine manipulation, meaning grasp shape and contact at the object. A wrist trajectory is the motion of each wrist recorded by a 6-DoF IMU at 200 Hz.

The site delivers all three together and keeps them distinct. This page claims for body pose only what the site states for body pose: 24 joints, 3D, in the annotations group. The per-joint visibility flag and the 21-joint count belong to hand pose, and the IMU rate belongs to the wrist streams. The second table sets the three side by side.

The site states that 3D hand and body pose are captured live on the rig rather than reconstructed after the fact, and lists body pose in the enrichment stage of its pipeline, with depth, hand pose, action segments and instance masks. The rig itself is a global-shutter head camera at 4K and 60 fps with a 120° horizontal field of view, two wrist cameras at 1920×1080 and 60 fps, a depth stream at 848×480 and 30 fps in metric millimetres, and 6-DoF IMUs on the head and both wrists at 200 Hz.

What the site does not state is how the 24 joints are derived: which sensors or estimation method produce them is not described on any page. That is set per program in the skill spec, and you should ask for it if your model depends on it. The third table lists the rig streams that travel with body pose.

The glossary states that body pose is captured synchronized with hand pose, depth and RGB under two milliseconds. The capture page describes the mechanism: the head-mounted stereo pair, both wrist cameras, the depth sensor and the IMUs run on one PTP clock domain, and every delivered stream carries int64 nanosecond timestamps in that domain.

The stated figures are a 1.12 ms median and 1.94 ms p99 against a 2.0 ms ceiling, and a stream that exceeds the ceiling for more than three consecutive frames is a reject reason. The frame rate of the body_joints stream itself is not stated and is set per program.

The episode schema is the canonical layout, and its annotations group lists body_joints; HDF5 and zarr mirror that layout. In the LeRobot v2.1 export, hand and body pose are flattened into a single observation.state vector, with 2×21×3 hand joints first and then 24×3 body joints, in that order. The guide on converting between RLDS and LeRobot says to keep that ordering and to check the vector against it.

RLDS names hand_joints as a step key, but body pose is not a named RLDS step key on its own. The WebDataset page lists hand_joints, action and contact_events for its annotation file and does not mention body joints. How body pose is laid out in those two formats, and in Rerun .rrd, is not stated, so name your loader in the spec. The fourth table summarizes this.

Body pose is part of the validated hour only through the checks the site defines for every episode: sync within the ceiling, calibration solved and within drift tolerance, coverage cells filled and consent documented. A separate accuracy gate for body joints is not stated, and no body pose accuracy figure is published.

On consent, the contributor agreement the site describes covers derivative representations, including hand and body pose estimates, and those layers sit under the same consent as the video. Delivered data ships under a perpetual, buyer-owned license. Rejected episodes are recaptured at the provider’s cost.

Several body pose details are not fixed on this site and are agreed in the skill spec. The last table separates what the site states from what is set per program, so you know what to put in writing before the quote.

Send the tasks, the environments and regions, and the delivery format you need, and name body pose in the streams and labels list. The site lists RGB as the baseline and the other streams as added where the spec calls for them, and says that hand pose, body pose, segmentation and action labels each add equipment, QA passes and cost, so order what the model consumes. If you have no written spec yet, the first draft is written with you on a call.

The order flow is spec, quote, Spec Pilot, then production. Where coverage for your tasks and regions is deep, the median time from signed spec to first validated batch is 48 hours; where it is thin, turnaround runs three to four weeks. There is no list price, and billing is per validated hour.