Guide
Using IMU Data in Robot Learning Datasets
Short answer
IMU data records acceleration and rotation over time, giving a policy motion the camera alone cannot see. Firsthand captures it from head- and wrist-mounted 6-DoF IMUs at 200 Hz, hardware-triggered onto the same PTP clock as RGB, depth, and hand pose. It ships as a synchronized imu stream in HDF5, zarr, RLDS, or WebDataset.
Last reviewed:
What is IMU data, and why does robot learning use it?
An inertial measurement unit (IMU) reports acceleration and angular velocity — how fast and in what direction a point is moving and rotating — on top of whatever a camera can see. A policy trained on video alone has to infer motion indirectly frame to frame; a policy trained with a synchronized IMU stream gets a direct, high-rate signal of how the head or hand actually moved during a manipulation, which helps disambiguate motion that a camera blurs, occludes, or samples too slowly to capture.
How does Firsthand capture IMU data?
IMU data comes from 6-DoF (accelerometer plus gyroscope) BMI088 sensors mounted at the head and on each wrist, sampled at 200 Hz. Like every other stream in the rig, the IMUs hardware-trigger onto the same PTP clock domain as the head and wrist cameras, depth, and hand pose, so an inertial sample and a video frame captured at the same instant carry matching timestamps rather than being aligned after the fact in software.
- 3× 6-DoF BMI088 IMU — head-mounted and dual wrist-mounted
- 200 Hz sample rate — faster than the camera frame rate, so multiple IMU samples fall within a single video frame
- Hardware-triggered onto the same PTP clock domain as RGB, depth, and hand pose
What does a delivered IMU stream actually look like?
IMU data ships as its own array or file per episode, sampled at its native 200 Hz rate rather than resampled down to the camera frame rate. Because the camera and IMU run at different rates, an episode carries more IMU samples than video frames for the same span of time.
| Field | What it carries |
|---|---|
| streams/imu_head | Head-mounted 6-DoF IMU, float32 [N, 6] at 200 Hz — HDF5 delivery |
| streams/imu_wrist_l | Left-wrist 6-DoF IMU, float32 [N, 6] at 200 Hz — HDF5 delivery |
| streams/imu_wrist_r | Right-wrist 6-DoF IMU, float32 [N, 6] at 200 Hz — HDF5 delivery |
| *_timestamp_ns | Per-sample hardware timestamp on the shared PTP clock domain |
| Sync | IMU, RGB, depth, and hand pose all hardware-triggered onto one clock |
N is the IMU sample count and differs from the video frame count T because the two streams run at different rates.
IMU samples run at 200 Hz while video runs at the camera frame rate, so consuming both streams together means resampling or windowing the IMU data to the frame you need rather than assuming a 1:1 sample-to-frame mapping.
When does a policy actually need IMU data?
IMU data matters most when a policy fuses vision with motion — for example distinguishing a fast, deliberate reach from a slow one, or recovering head or wrist motion during fast manipulation that a camera frame blurs or samples too slowly to resolve. Tasks that depend only on static scene understanding, such as classification or coarse navigation, are often fine without it.
FAQ
Using IMU Data in Robot Learning Datasets, answered.
01What kind of IMU does Firsthand use?
3× 6-DoF (accelerometer plus gyroscope) BMI088 IMUs — one head-mounted, one on each wrist — sampled at 200 Hz.
02Is IMU data synchronized with the video and depth streams?
Yes. IMU, RGB, depth, and hand pose all hardware-trigger onto the same PTP clock domain, so IMU samples and video frames captured over the same span carry matching timestamps rather than being aligned in software after capture.
03What formats does IMU data ship in?
HDF5 and zarr carry it natively as its own array (streams/imu_head, streams/imu_wrist_l, streams/imu_wrist_r). RLDS and WebDataset deliveries carry the same per-sample timestamped structure adapted to each format's convention.
04Why does an episode have more IMU samples than video frames?
The IMU samples at 200 Hz, which is faster than the camera frame rate, so multiple IMU samples fall within the span of a single video frame. This is expected, not a data quality issue.
05Do all robot learning tasks need an IMU stream?
No. IMU data matters most where a policy fuses vision with motion, such as fast or occluded manipulation. Tasks relying mainly on static scene understanding are often fine without it.
Sources
Where is this documented?
- RLDS — Reinforcement Learning Datasets format (opens in a new tab)The episode format used for RLDS delivery, including per-step observation keys for auxiliary sensor streams. github.com
- WebDataset (opens in a new tab)Streaming tar-based format used for sharded sensor-stream delivery. github.com
Check it against the sample pack.
40 episodes across 4 environments, delivered in the exact schema these guides describe.