Glossary
What Is Gaze Tracking in Egocentric Video?
Last reviewed: October 2026
Short answer
Gaze tracking in egocentric video is a record of where the wearer was looking while a task was filmed from their own point of view. Firsthand’s episode schema lists it as the gaze_target field, delivered beside head video, IMU, hand pose and action segments on one clock when the skill spec calls for it.
01
What is gaze tracking in egocentric video?
Egocentric video shows what a wearer’s head camera sees. Gaze tracking adds which part of that view the wearer was attending to. A head-mounted camera moves with the head, so its frame is the eye-line view, but the eyes can rest anywhere inside it; a gaze record says where. The site describes gaze as a signal about intent: where a person looks tends to lead where the hands go. It does not state a measured improvement from adding gaze to a training mix, so none is quoted here.
02
What gaze data does Firsthand state it delivers?
Firsthand’s episode schema lists gaze_target in the annotations group: where attention was directed, where the skill spec calls for it. The docs layout shows each entry as a timestamp and an object id. It travels beside head video at 60 fps, the head IMU at 200 Hz in 6-DoF, hand joints, contact events and action segments, all in int64 nanoseconds on one PTP clock domain. The site names no eye-tracking device, gaze sampling rate, accuracy figure or calibration error, and this page makes none of those claims. Whether a delivery includes gaze is a spec decision, not a default.
03
How does gaze data differ from head direction?
Gaze is where attention was directed; head direction is where the head pointed. The head IMU records how the head moved, at 200 Hz in 6-DoF, and the head camera shows the eye-line view. Neither says which object inside that view the wearer looked at. gaze_target does.
Gaze is also not hand pose, the per-joint 3D position of 21 joints per hand, or an action segment, a verb and noun with start and end times. Because every field sits on one timeline, a model can read what the wearer looked at against what the hands then did.
04
How is gaze data used in robot learning?
The site’s guidance is conditional: it recommends the head-rig tier for tasks where head orientation and gaze matter, and adds gaze target when a skill spec calls for it. The capture tier is chosen to match the spec.
The site states no gaze-conditioned policy result, benchmark gain or recommended gaze-to-hand mix. Robot results are Not stated.
05
What fields and formats carry gaze data?
Every format is exported from the same QA-passed episodes: RLDS, LeRobot, WebDataset and HDF5 / zarr. To read gaze_target directly, ask for HDF5 or zarr, whose annotations/ group mirrors the episode schema. For RLDS, LeRobot and WebDataset the site states how hand joints and actions are carried but not a gaze key, so where gaze sits is set in the skill spec.
06
What do public egocentric datasets report about gaze?
The Ego4D paper states that portions of its video are accompanied by eye gaze, and its Table 1 lists 45 hours of gaze against 3,670 hours of RGB video. That describes that dataset only, not Firsthand, but it shows gaze is usually a subset modality, which is why it is worth naming in a spec.
07
How do you request a custom gaze dataset?
Write down the tasks, environments and objects your model needs, say whether gaze is in scope and how you expect to read it, and name the delivery format. Most orders begin with a Spec Pilot: a small, fast first batch that proves the spec on real footage before you scale. This is on-demand custom collection, not a catalog download.
Paid, consented contributors perform the task; Firsthand does not capture at a customer’s own facility. Faces, plates and screens in frame are irreversibly blurred at ingest, with a reviewer confirming. Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks. Billing is per validated hour, with no list price.
08
Gaze vs head video, head IMU and hand pose: what each records
| Data type | What it records | Delivery field | Stated spec |
|---|---|---|---|
| Gaze target | Where attention was directed, where the spec calls for it | annotations/gaze_target | list[{t_ns, object_id}]; sampling rate Not stated |
| Head video | The eye-line first-person view | streams/head_rgb | 60 fps |
| Head IMU | How the head moved | streams/imu_head | 200 Hz, 6-DoF |
| Hand pose | Per-joint 3D hand skeleton | annotations/hand_joints | float32 [T, 2, 21, 3], metres, head-camera frame |
| Action segments | Verb and noun with start and end times | annotations/action_segments | list[{start_ns, end_ns, verb, noun}] |
10
Gaze details: stated on the site vs Not stated
| Detail | Status |
|---|---|
| A gaze_target field exists | Stated: annotations group of the episode schema |
| Entry layout | Stated in the docs layout: a timestamp and an object id |
| Pixel coordinates or gaze direction | Not stated |
| Eye-tracking hardware | Not stated |
| Sampling rate and frame alignment | Not stated; set per program in the skill spec |
| Accuracy or calibration error | Not stated |
| Gaze key in RLDS, LeRobot, WebDataset | Not stated |
| Included in a given delivery | A spec decision, confirmed on the Spec Pilot batch |
11
Frequently asked questions
- What is gaze tracking in egocentric video?
- A record of where the wearer was looking while a task was filmed from their own point of view. On Firsthand’s episode schema it is the gaze_target field in the annotations group.
- Is gaze the same as head direction?
- No. The head IMU records how the head moved, and the head camera shows the eye-line view. gaze_target records where attention was directed inside that view.
- Does Firsthand state which eye tracker it uses?
- No. The site lists a gaze_target field but names no eye-tracking device, sampling rate, accuracy figure or calibration error. Those are set per program in the skill spec.
- Is gaze delivered with every episode?
- Not automatically. Whether a delivery includes gaze_target is a spec decision. Name it in the skill spec and confirm it on the Spec Pilot batch.
- Which format should I choose to read gaze?
- HDF5 or zarr, since its annotations/ group mirrors the episode schema. Where gaze sits in RLDS, LeRobot or WebDataset is set per program in the skill spec.
- How fast can a custom gaze dataset be collected?
- Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks. There is no list price: you are quoted against your spec.
Related terms