Delivery format
Egocentric episode schema: every field explained
The canonical layout every delivery format projects from — meta, streams, annotations, calibration.
By the Firsthand capture teamLast updated September 23, 2026
Short answer
The episode schema is the canonical layout every Firsthand delivery format is a projection of. Each episode has four groups — meta, streams, annotations and calibration — with explicit dtypes, shapes and metric units. Nothing in it is optional or upsold; the sample pack ships in this exact schema.
Field mapping
How Firsthand fields map into this format
- meta.json
- episode_id, skill_id, site_id, participant_id, duration_ns, license, reject_flags
- streams/
- head_rgb, wrist_l/r, depth_mm, depth_conf, imu_head, imu_wrist_l/r, *_timestamp_ns
- annotations/
- hand_joints, hand_visible, body_joints, contact_events, instance_masks, objects, action_segments, gaze_target
- calibration/
- intrinsics, extrinsics (SE3), rms_reproj_px
- Units
- Metric — depth in millimetres, hand joints in metres
Loading
Loading it
EP_0117_KITCHEN_A/
├── meta.json episode_id, skill_id, site_id, duration_ns, license
├── streams/
│ head_rgb uint8 [T, 2160, 3840, 3] 60 Hz
│ wrist_l/r uint8 [T, 1080, 1920, 3] 60 Hz
│ depth_mm uint16 [T, 480, 848] 30 Hz # metric mm
│ imu_* float32 [N, 6] 200 Hz
│ *_timestamp_ns int64 per stream, PTP domain
├── annotations/
│ hand_joints float32 [T, 2, 21, 3] # metres, head frame
│ action_segments list[{start_ns, end_ns, verb, noun}]
│ instance_masks uint16 [T, 480, 848]
└── calibration/
intrinsics, extrinsics (SE3), rms_reproj_px < 0.28Gotchas
What to watch for
- T is head-camera frames; N is IMU samples — they differ because streams run at different rates.
- Streams are stored at native rate and resampled by the loader, never in the delivery.
- The head-camera frame is the identity for all extrinsics.
Get a Episode schema sample.
40 episodes across 4 environments, delivered in the format your stack already reads.