Guide
Why Camera Synchronization Matters in Robotics Data
Short answer
Camera synchronization matters because a policy learns from where sensors say the hand and object are at time t — if RGB, depth, IMU and pose are offset by even tens of milliseconds during a fast reach, that supervision is wrong exactly when the motion is most informative. Synchronized multimodal capture is what makes multi-stream egocentric data trainable rather than merely co-recorded.
Last reviewed:
Why does synchronization matter for training data?
A manipulation policy is trained on the assumption that its input streams describe the same instant. If depth lags video, or IMU drifts against pose, the model learns a slightly wrong correspondence between what it sees and what actually happened — and that error is worst exactly during fast contact and grasp, the moments with the most signal. Synchronized multimodal capture is the foundation every other annotation (action segments, hand pose, skill spec coverage) depends on.
How is multi-camera synchronization achieved?
The fix is a shared hardware clock, not software timestamps. PTP (Precision Time Protocol, IEEE 1588) gives every sensor in a rig sub-microsecond clock alignment over the network, and hardware-triggered exposures mean each frame is captured on that shared timebase rather than whenever each device happens to fire.
- Software timestamps (loosely synced)
- Each device stamps its own clock; drift and jitter accumulate over a session
- PTP clock domain, hardware-triggered
- One shared timebase; exposures fire together, so residual offset is small and measurable
- Firsthand delivery
- int64 nanosecond timestamps per stream, measured residual median 1.12 ms, p99 1.94 ms, 2.0 ms reject ceiling
Comparing sync approaches by what they leave to chance.
How is sync error actually measured?
Synchronization is a claim you can verify, not just trust. Firsthand measures the residual offset between streams against a periodic strobe visible to the cameras and detectable in the IMU, per episode, and publishes the reject log for any episode that breaches the 2.0 ms ceiling. The full measurement methodology is in the sync-error guide.
FAQ
Why Camera Synchronization Matters in Robotics Data, answered.
01What happens if cameras are not synchronized during capture?
The model learns from a hand position, depth reading and image that do not actually correspond to the same instant. The resulting supervision is wrong precisely during fast reach and grasp motions — the moments with the most training signal.
02What is PTP clock sync?
PTP (Precision Time Protocol, IEEE 1588) synchronizes clocks across devices to sub-microsecond accuracy over a network, giving every sensor in a rig a shared timebase so their timestamps are directly comparable.
03What is cross-stream sync error?
It is the residual timing offset between streams meant to be simultaneous — RGB, depth, IMU and pose — measured in milliseconds after clock alignment and hardware triggering.
04How small does sync error need to be?
Firsthand rejects any episode with a residual sync error above 2.0 ms; delivered episodes measure median 1.12 ms and p99 1.94 ms.
05Can I verify synchronization myself on delivered data?
Yes. HDF5 and Rerun exports carry the per-stream nanosecond timestamps, so you can recompute the residual directly rather than trust a published number.
Sources
Where is this documented?
- IEEE 1588 (PTP) — IEEE Standards Association (opens in a new tab)Precision Time Protocol standard standards.ieee.org
Check it against the sample pack.
40 episodes across 4 environments, delivered in the exact schema these guides describe.