Egocentric Video with Wrist Trajectories

Egocentric video with wrist trajectories is first-person footage of a task, recorded from a head-mounted camera, delivered together with the motion of the wearer’s wrists over time. The wrist trajectory is the coarse path of the hand: where it went, how fast, and how it was oriented on the way to an object.

The site is specific about how that signal is produced. Firsthand records wrist-level motion as 6-DoF IMU streams, streams/imu_wrist_l and streams/imu_wrist_r, at 200 Hz, one IMU on each wrist. The site describes these streams as giving the coarse trajectory that finer hand-joint detail sits on top of. Hand joints, contact events and action segments are delivered alongside.

This page is for robot learning, VLA and humanoid teams that want to specify those fields and order them collected, fast and on demand, as a custom collection.

A policy trained from human demonstration needs an action signal, and human footage has no robot joint log. The site’s pretraining guide explains that the action has to be represented some other way, and that for egocentric video the two stand-ins are wrist-level pose and hand pose.

Wrist-level motion is also the more portable of the two. The guide notes that a wrist label does not encode any one gripper’s joint layout, so the same trajectory can act as a coarse target for a human arm, a parallel-jaw gripper or a dexterous hand. For fine manipulation, finger-level hand pose matters more, so a spec should state both requirements rather than assume one implies the other.

The second reason is provenance. NVIDIA’s EgoScale represents action as wrist-level 6-DoF motion plus hand pose and, per the site’s guides, produced those labels after capture with off-the-shelf SLAM and hand-pose estimation across a 20,854-hour corpus. Firsthand records labels on a synchronized rig during capture instead. The site treats both as legitimate.

Wrist motion comes from three Bosch BMI088 IMUs running at 200 Hz on the instrumented rig, one on the head and one on each wrist. Each is 6-DoF, bias-calibrated and hardware-timestamped on the same clock domain as the cameras and depth stream, writing int64 nanosecond timestamps.

Hand joints are tracked live against the captured depth stream, per hand, and stored as annotations/hand_joints: 21 joints per hand, in metres, in the head-camera frame, as float32 with shape [T, 2, 21, 3]. A per-joint hand_visible flag records occlusion rather than estimating through it. Two wrist-mounted cameras at 1920×1080 and 60 fps add a close optical view of each hand; they are a separate optional stream, covered below.

The site does not state how a wrist pose is computed from the IMU streams, which coordinate frame the wrist motion uses, or any accuracy, smoothing or filtering figure. Those details are set per program in the skill spec.

The field table lists what travels with the wrist streams, using the names and shapes the episode schema states. A dexterous-hand program will normally add hand joints and visibility.

Action segments use a taxonomy of 97 verbs and 512 nouns, with start and end to the frame, human-reviewed. Contact events are recorded on grasp and release alongside hand pose and depth.

Three nearby pages are easy to confuse with this one. A wrist trajectory is motion, not a picture: it comes from the wrist IMUs. Wrist camera data is the optical stream from the two wrist-mounted cameras, which show the hand and the object up close but are not a trajectory. Hand tracking data is the 21-joint skeleton of each hand, which describes finger shape in the head-camera frame.

In practice these combine. The differentiation table shows what each covers, so you can specify the fields you need rather than discover a gap later.

Every sensor runs in one PTP clock domain and exposures are hardware-triggered. The site states a 1.12 ms median offset, 1.94 ms at the 99th percentile and a 2.0 ms ceiling, measured as the absolute timestamp offset between the head camera and every other stream.

The streams run at different rates: head and wrist cameras at 60 fps, depth at 30 fps and the IMUs at 200 Hz. Wrist IMU samples therefore outnumber video frames for the same span, and the site advises resampling or windowing the IMU to the frame you need rather than assuming a one-to-one mapping. The head-camera frame count T indexes hand_joints, while action segments are start_ns and end_ns pairs on the same timeline.

All formats are exported from the same QA-passed episodes as projections of the canonical episode schema, so switching formats does not change what a wrist or hand label contains. The format table lists what the site states for each.

If you plan to work with the wrist IMU streams directly, ask for HDF5 or zarr: its streams/ and annotations/ groups mirror the episode schema and carry per-stream nanosecond timestamps. For the other formats, the site states how hand joints and actions are carried but not a dedicated wrist IMU key, so where the wrist signal sits in an RLDS, LeRobot or WebDataset delivery is set per program in the skill spec.

Quality is checked at the episode level, and the gates are machine-checked at ingest. They are a sync breach beyond the 2.0 ms ceiling for more than three consecutive frames, calibration drift beyond tolerance, the manipulated object out of frame for more than 15 percent of the episode, more than 35 percent invalid depth pixels in the manipulation region, an undocumented bystander, and a mismatch against the named skill spec.

Calibration brackets the session: per-camera reprojection must stay under 0.28 px RMS, and drift is written into the episode as calib_drift_mm. A human reviewer then checks each episode against the task spec. Rejected episodes go to the reject log, which ships with every batch, and are recaptured at our cost. Only validated hours are billed.

The site states no accuracy or error figure for wrist trajectories specifically, so none is quoted. Any tolerance on the wrist signal is set per program in the skill spec and confirmed on the first batch.

Every custom collection starts with a written skill spec: the tasks, the setting, the object set, the capture tier, the annotation depth and the accept criteria, signed before capture begins. For a wrist trajectory order it also names which wrist and hand fields you need, the delivery format you load from, and every detail the site does not state, as the final table shows.

Paid, consented contributors perform the real task against your spec while wearing the rig. We do not capture robot or teleoperation data, and we do not capture at a customer’s own facility. Every episode is tied to a signed contributor release, and faces, plates and screens in frame are detected and irreversibly blurred at ingest, with a reviewer confirming the result.

The dataset ships under a perpetual, irrevocable, worldwide commercial licence that is non-exclusive by default, with the buyer owning the resulting dataset and exclusivity available. Where coverage is deep, the median time from signed spec to first validated batch is a median of 48 hours; where it is thin, turnaround runs three to four weeks while recruiting expands in parallel. Billing is per validated hour, not per raw hour recorded.

Write down the tasks, environments and objects your model needs, list the wrist and hand fields you want, and name the delivery format. Most orders begin with a Spec Pilot: a small, fast first batch that proves the spec and accept criteria hold on real footage before you scale.