Guide
How to measure cross-stream sync error in egocentric capture
Short answer
Cross-stream sync error is the residual timing offset between streams meant to be simultaneous — RGB, depth, IMU and pose. Firsthand hardware-triggers every sensor onto one PTP clock domain, writes int64 nanosecond timestamps per stream, and measures the residual: median 1.12 ms, p99 1.94 ms, with a 2.0 ms reject ceiling.
Why does sync error matter?
If depth lags video by tens of milliseconds during a fast reach, the supervision is wrong exactly when the motion is interesting. Sub-frame synchronization is what makes multi-stream egocentric data trainable rather than merely co-recorded. It is the foundation every other annotation depends on.
How is the residual measured?
Every sensor shares one PTP (IEEE 1588) clock domain and exposures are hardware-triggered, so each frame carries a timestamp on the same timebase. We measure the residual offset between streams against a periodic strobe visible to the cameras and detectable in the IMU, per episode.
- Clock
- One PTP domain, hardware-triggered exposures
- Timestamps
- int64 nanoseconds, written per stream
- Median residual
- 1.12 ms
- p99 residual
- 1.94 ms
- Reject ceiling
- 2.0 ms — any episode over it is discarded
Measured across delivered episodes; the reject log records breaches.
Can I verify sync on my own copy?
Yes. The HDF5 and Rerun exports carry the per-stream nanosecond timestamps, so you can recompute the residual yourself rather than trust our number. The Rerun recording plots sync error on its own timeline next to the camera and hand entities.
from firsthand_ego import Episode
import numpy as np
ep = Episode.open("data/EP_0117_KITCHEN_A")
# per-stream PTP timestamps, in nanoseconds
head = ep.timestamps("head_rgb")
depth = ep.timestamps("depth_mm")
# nearest-neighbour residual, ms
res = np.abs(depth[:, None] - head[None, :]).min(axis=1) / 1e6
print(f"median {np.median(res):.3f} ms p99 {np.percentile(res, 99):.3f} ms")
assert res.max() < 2.0Check it against the sample pack.
40 episodes across 4 environments, delivered in the exact schema these guides describe.