What Is Sensor Fusion in Egocentric Data Capture?

No single sensor sees everything a manipulation task needs. A monocular view leaves depth ambiguous, and a blurred or occluded frame still has a real trajectory that an inertial sensor records directly. Sensor fusion brings complementary streams together so each one covers another’s weak point. It has two halves. The capture half records the streams together, on one clock, in a shared coordinate frame. The model half is how a team’s own pipeline combines them. This page covers the capture half. Whether Firsthand delivers any fused output, or runs a fusion algorithm, is Not stated. What it does state is a set of aligned, timestamped streams, which is the input any fusion step depends on. Aligned streams are also what the multimodal training data entry describes.

Firsthand states a head-mounted global-shutter 4K camera at 60 fps with a 120° lens, two wrist cameras at 1080p and 60 fps, depth at 848 × 480 and 30 fps, and 6-DoF IMUs at 200 Hz, one on the head and one per wrist. Stereoscopic video comes from a head-mounted stereo rig at a human interpupillary baseline, hardware-synced on one clock with depth, IMU and hand pose. All streams are hardware-triggered onto a single PTP clock domain. Intrinsics and extrinsics are re-solved with a checkerboard target at the start and end of every session, and an episode ships only if RMS reprojection stays under 0.28 px. Collection is fast, on demand and to a written spec.

Fusion assumes that a sample from one sensor describes the same instant as a sample from another. If depth lags video by tens of milliseconds during a fast reach, the combined signal is wrong exactly when the motion is interesting, as the cross-stream sync error entry puts it. Alignment has two parts. A shared clock makes timestamps comparable: IEEE 1588 describes PTP as precise clock synchronization in packet-based networked systems, supporting the sub-microsecond range. Hardware triggering makes the exposures start together. Firsthand states both, then writes int64 nanosecond timestamps per stream.

The site reports a measured residual of 1.12 ms median and 1.94 ms at the 99th percentile, with a 2.0 ms ceiling above which an episode is rejected. The streams run at different native rates, 60 fps, 30 fps and 200 Hz, so the per-stream timestamps are the join key. The loader, not the delivery, resamples between them.

Stereo video recovers geometry. Two horizontally offset views give disparity, and the site states that 3D hand joints, object poses and contacts are annotated against the stereo geometry rather than guessed from a single frame. What stereo does not give is a motion record between frames. The IMU does: acceleration and angular velocity sampled at 200 Hz, more than three times the 60 fps video rate, hardware-timestamped on the same clock.

Each covers what the other lacks. Video and stereo see the scene but sample it in discrete frames. The IMU feels motion continuously but cannot see the scene. Fusing them is the buyer’s step. A head or camera pose trajectory built from the two is not something the site states it delivers, and no tolerance or accuracy for such an estimate is stated.

Start with the clock. Ask whether every stream carries timestamps on one common timebase, and whether a sync residual is measured and reported against a stated rejection rule. Firsthand’s HDF5 and Rerun exports carry per-stream nanosecond timestamps, so the residual can be recomputed on your own copy.

Then check the geometry. Ask how calibration was solved, how often, and what happens if it drifts. Firsthand states two solves per session and a 0.28 px reprojection limit. Finally, ask for the rate of every stream separately and which streams an order actually carries, because the spec decides that. Anything a listing does not state, such as the numeric stereo baseline, should be treated as Not stated until confirmed in writing.

Request a custom dataset and describe the skills, environments and streams you need. Collection is fast, on demand and to a written spec, and the site states that hardware triggering is how the rig works rather than an optional add-on. Deliveries use RLDS, LeRobot format or WebDataset, with HDF5, zarr and Rerun exports carrying per-stream timestamps, under a buyer-owned commercial license. A different sensor layout, trigger rate or stream count is not stated, so confirm it in the spec conversation.

What is sensor fusion in egocentric data capture?

Combining streams from several sensors, such as stereo video, depth and IMU, into one time-aligned record of the same moment. It depends on the streams sharing a clock.

Does Firsthand deliver fused data or run a fusion algorithm?

Not stated. The site states aligned, timestamped streams on one PTP clock domain, which is the input a fusion step needs. The fusion itself is the buyer’s pipeline.

Which sensors does Firsthand combine in one episode?

A head-mounted global-shutter 4K camera, two wrist cameras, depth, 200 Hz 6-DoF IMUs and 3D hand pose, with stereoscopic video from a head-mounted stereo rig, all hardware-triggered onto one PTP clock domain.

How well are the streams synchronized?

The site reports a 1.12 ms median and 1.94 ms p99 cross-stream residual, with any episode over the 2.0 ms ceiling rejected.

Why do stereo video and IMU data work well together?

Stereo recovers scene geometry from disparity, and the IMU records continuous motion at 200 Hz. Each covers what the other cannot see or sample.

Does Firsthand deliver a head pose or SLAM trajectory?

The site does not state that it does. The streams are delivered with calibration and per-stream timestamps so a team can build its own estimate.

Can I check the synchronization myself?

Yes. The HDF5 and Rerun exports carry per-stream nanosecond timestamps, so the residual can be recomputed on your own copy.

How do I order fast, on-demand multi-sensor collection?

Request a custom dataset with your skills, environments and streams. Collection is fast, on demand and to a written spec.