Time-Synchronized Egocentric Sensor Data

Time-synchronized egocentric sensor data is first-person task footage recorded together with the other sensor streams of the same moment, with every stream stamped against one shared clock. On Firsthand’s rig those streams are a head camera, two wrist cameras, a depth stream and IMUs on the head and both wrists.

The key word is synchronized. Several sensors recording at once is not the same as sensors recording the same instant. Firsthand hardware-triggers the streams onto one PTP clock domain instead of lining up independent device clocks afterwards, and publishes the residual offset so you can judge it.

A manipulation policy is trained on the assumption that its inputs describe one instant. The site’s synchronization guide puts the failure plainly: if depth lags video by tens of milliseconds during a fast reach, the supervision is wrong exactly when the motion is most informative. The model learns a slightly wrong correspondence between what it sees and what happened, and the error is worst during contact and grasp.

Synchronization also underpins everything layered on top. Hand pose, action segments and masks are labelled against the streams, so if the streams disagree about when something happened, every label inherits the disagreement.

For a buyer this becomes a procurement question. Not whether there are several streams, but how close they are, how that is measured, and what happens when they are not close enough. The sections below answer each with figures the site already states.

The table below lists the streams on the instrumented head-mounted rig with the rates and resolutions the site states. The free sample pack, for reference, carries 8 synchronized streams per episode.

Hand pose and action segments are annotations rather than sensors. They are produced from the captured streams and carry timestamps aligned to every other stream; hand pose is delivered at 21 joints per hand. Not every order needs every stream. Depth, IMU and hand pose are added where the written spec calls for them, and a buyer who needs only RGB and action labels can order just that.

A lighter tier exists: a contributor-worn 360 camera on a head strap. It gives up depth, wrist views and sub-millisecond sync in exchange for reach, so it is not the rig this page describes.

PTP, the Precision Time Protocol defined in IEEE 1588, synchronizes clocks across devices over a network to sub-microsecond accuracy, as the site’s glossary defines it. In a capture rig it gives every sensor a shared timebase, so timestamps are directly comparable rather than each device keeping its own clock.

Firsthand goes a step beyond clock alignment. A PTP grandmaster and a hardware trigger line reach every camera and IMU, every sensor runs in one PTP clock domain, and exposures are hardware-triggered, so the sensors are told to capture together. The site contrasts this with software timestamps, which drift and jitter over a session. Each stream then writes int64 nanosecond timestamps in the PTP domain.

The stated residual is in the second table. The median is the absolute timestamp offset between the head camera and every other stream, measured over 12,480 frames. A median and a p99 describe the typical and the near-worst delivered episode; the 2.0 ms ceiling is the gate. The site states no drift rate, clock model or timestamp resolution beyond int64 nanoseconds, so none are claimed here. Those details are set per program in the skill spec.

Synchronization is a claim you can verify. Firsthand measures the residual offset between streams against a periodic strobe that is visible to the cameras and detectable in the IMU, per episode. Because every stream records the same strobe, any gap between where each stream places it is the residual.

IMUs are also aligned during calibration with a static hold plus a figure-eight excitation, and the stereo rig that produces depth is validated in the same pass. Calibration brackets the session: intrinsics and extrinsics are solved before and after capture, per-camera reprojection must stay under 0.28 px RMS, and the open-to-close drift is written into the episode as calib_drift_mm. Time and geometry are checked together.

You do not have to take the number on trust. The HDF5 export carries per-stream nanosecond timestamps, and the Rerun recording plots sync error on its own timeline beside the camera and hand entities, so you can recompute the residual on your own copy.

The 2.0 ms ceiling is a gate, not a target. The glossary states that any episode breaching it is rejected, and the ingest criteria state it as a sync breach beyond the ceiling for more than three consecutive frames. Both are machine-checked at ingest alongside calibration drift beyond tolerance, the manipulated object out of frame for more than 15 percent of the episode, more than 35 percent invalid depth pixels in the manipulation region, an undocumented bystander, and a mismatch against the named skill spec.

A human reviewer then checks each episode against the task spec. Rejected episodes go to the reject log, which ships with every batch, and are recaptured at our cost rather than delivered with a caveat. Only footage that clears the accept criteria is billed as a validated hour, so a sync failure is a cost to us, not to you.

Synchronized does not mean identical rates. The head camera and wrist cameras record at 60 fps, depth at 30 fps and the IMUs at 200 Hz. Synchronization means every sample carries a timestamp on the same clock, so a video frame, a depth frame and the IMU samples nearest to them can be matched by time.

Because the IMUs sample faster than the cameras, an episode holds more IMU samples than video frames for the same span. The site advises resampling or windowing the IMU to the frame you need rather than assuming a one-to-one mapping, and the same care applies to depth at 30 fps against 60 fps video. Action segments are stored as start_ns and end_ns pairs with a verb and noun, so their boundaries sit on the same nanosecond timeline as the streams they label.

The third table shows how each delivery format carries time, as the site states it. All are exported from the same QA-passed episodes, so sync is checked once against the canonical episode rather than once per format.

If you plan to verify synchronization yourself, ask for HDF5. WebDataset ships tar shards of 512 samples with per-sample keys head.jpg, wrist_l.jpg, wrist_r.jpg, depth.png and ann.json. The site states no timestamp key for WebDataset samples, so how timestamps are carried in a WebDataset delivery is set per program in the skill spec.

Every custom collection starts with a written skill spec: the tasks, the setting, the object set, the capture tier, the annotation depth and the accept criteria, signed before capture begins. For a synchronized order the spec names the streams you need, the delivery format you load from, and any sync tolerance you want the batch held to.

Paid, consented contributors perform the real task against your spec while wearing the rig. We do not capture robot or teleoperation data, and we do not capture at a customer’s own facility. Every episode is tied to a signed contributor release, and faces, plates and screens in frame are detected and irreversibly blurred at ingest, with a reviewer confirming the result.

The dataset ships under a perpetual, irrevocable, worldwide commercial licence that is non-exclusive by default, with the buyer owning the resulting dataset and exclusivity available. Where coverage is deep, the median time from signed spec to first validated batch is a median of 48 hours; where it is thin, turnaround runs three to four weeks while recruiting expands in parallel. Billing is per validated hour, not per raw hour recorded.

Write down the tasks, environments, objects and annotation depth your model needs, list the streams you want synchronized, and name the delivery format. Most orders begin with a Spec Pilot: a small, fast first batch that proves the spec and accept criteria hold on real footage before you scale.