Egocentric Video With Depth and IMU Data

A standard egocentric collection captures first-person RGB video from head- and wrist-mounted cameras. Adding depth and IMU extends that bundle with a metric depth stream and inertial motion data, both hardware-triggered onto the same clock as the video, so a training pipeline receives distance and motion signals genuinely aligned to the frame rather than a separate recording that merely happened at roughly the same time.

This is distinct from a standalone depth sensor or a loosely time-stamped IMU log: every stream in the bundle shares one PTP clock domain with per-stream timestamps, which is what makes the depth, IMU, RGB, and hand pose usable together rather than individually.

RGB alone tells a policy what a scene looks like but not how far away anything is or how the camera and hands are actually moving through space. Depth gives the metric distance to every pixel, which matters for grasp planning, obstacle avoidance, and any task where a policy has to judge reach or clearance. IMU gives the fine-grained motion trajectory of the head and wrists between video frames, which is a denser signal than frame-to-frame visual motion alone can provide.

Which streams a given order needs depends on the policy being trained: a perception model may need depth more than IMU, while a manipulation policy learning from wrist motion may lean on IMU alongside hand pose. Both are specified per order rather than bundled as an all-or-nothing default.

Depth is recovered from the calibrated stereo pair already on the rig rather than a separate active depth sensor, which is why it stays accurate outdoors and indoors alike — active depth sensors commonly struggle in bright outdoor light. It is delivered as a depth_mm stream in metric millimetres alongside a depth_conf per-pixel confidence channel, registered to the head-camera frame using the solved calibration extrinsics.

The rig’s calibration — intrinsics and extrinsics, solved as SE3 — is shipped with the episode rather than assumed, so depth can be re-derived independently if a buyer’s pipeline needs to. The site describes the stereo baseline only qualitatively, not as a fixed numeric spec.

Three 6-DoF (accelerometer plus gyroscope) BMI088 sensors — one head-mounted and one on each wrist — sample at 200 Hz, faster than the camera frame rate, and hardware-trigger onto the same PTP clock domain as the cameras and depth. Because the IMU samples more often than the cameras capture frames, an episode carries more IMU samples than video frames for the same span of time; consuming the two streams together means resampling or windowing the IMU data to the frame needed, rather than assuming a 1:1 sample-to-frame mapping.

Each IMU stream is delivered as its own array — streams/imu_head, streams/imu_wrist_l, streams/imu_wrist_r — so the head and each wrist can be analyzed independently or together.

Every active stream is hardware-triggered onto a shared clock with per-stream timestamps, rather than merely co-recorded and aligned afterward in software. Calibration for the sensor pair that benefits most from a tight time alignment — the stereo rig producing depth — is validated during the calibration pass; the IMUs are aligned during calibration via a static hold plus a figure-eight excitation.

The result is that an observation at a given step — RGB frame, depth frame, hand pose, and the IMU samples closest to that instant — is actually aligned, which matters most for any downstream task that fuses more than one stream rather than consuming them independently.

Depth is carried as streams/depth_mm (uint16 [T, 480, 848] at 30 Hz, metric millimetres) and streams/depth_conf in HDF5 and zarr delivery, as observation.depth_mm in RLDS, and as depth.png (16-bit, decode as uint16 millimetres, not 8-bit) in WebDataset. LeRobot’s mp4-based layout does not embed depth in the video file itself, so a policy consuming depth pulls it from the accompanying HDF5 export.

IMU is carried as streams/imu_head, streams/imu_wrist_l, and streams/imu_wrist_r — float32 [N, 6] at 200 Hz — in HDF5 and zarr, with RLDS and WebDataset carrying the same per-sample timestamped structure adapted to each format’s own convention. Which fields a delivery actually contains depends on which streams the written spec ordered.

Every episode passes through calibration and synchronization checks on every active stream, not just RGB — depth and IMU are checked for alignment to the shared clock alongside hand pose, and a pass/reject decision is made per episode. Episodes that fail are rejected and, where it matters to the buyer’s coverage, recaptured rather than delivered with a quality caveat, and a reject log accompanies delivery.

For an order that includes depth and IMU specifically, QA also confirms those streams are correctly aligned to the video and to each other, since a depth frame or IMU sample that merely sits in the same episode without verified timestamp alignment does not pass the same bar as streams confirmed to line up.

A collection starts with a written spec covering the tasks, scenes, and viewpoint, plus which streams the policy needs — RGB alone, or RGB with depth, IMU, and hand pose added. The spec also sets delivery format: RLDS, LeRobot, WebDataset, HDF5, zarr, or raw streams with metadata.

A Spec Pilot validates the plan on a small batch before the full order runs, so stream coverage, sync quality, and label fidelity can be checked before they are repeated at volume. Delivery ships under a perpetual, buyer-owned commercial license.