Egocentric Video with Eye Gaze Data

Egocentric video with eye gaze data is first-person footage of a task, recorded from a head-mounted camera, delivered together with a record of where the wearer’s attention was directed. The viewpoint is the eye-line view; the gaze annotation says which part of that view the wearer was attending to.

The site is specific about the field and quiet about the instrument. Firsthand’s episode schema lists gaze_target in its annotations group, and the site describes it as where attention was directed, where the spec calls for it. It does not state an eye-tracking device, a gaze sampling rate or an accuracy figure, so this page makes none of those claims.

This page is for robot learning, VLA and humanoid teams that want to specify gaze fields and order them collected, fast and on demand, as a custom collection.

Gaze is a signal about intent: where a person looks tends to lead where the hands go. The site’s collection pages recommend the head-rig tier when gaze matters, because it matches the eye-line viewpoint and adds IMU data for tasks where gaze and head motion are part of what the model needs to learn.

The site also notes that when footage is anonymized, the hands, objects, gaze and scene are the signal that must be left untouched, which is why a consented operator and their point of view stay intact in delivered data.

The site does not state a measured benefit from adding gaze to a training mix, so none is quoted here. Whether gaze is worth the added equipment and review for your policy is a decision to test on a small first batch.

The site’s action-labels entry states that Firsthand captures action labels directly at the time of recording rather than inferring them afterward, and lists gaze target among them with hand pose, wrist trajectories, contact events, body joints and action segments, annotated as part of the same capture session as the egocentric video and depth.

The site does not state which sensor or method produces gaze_target, whether the rig carries eye-tracking cameras, or how often the gaze field is sampled. Those details are set per program in the skill spec and confirmed on the first batch.

The field table lists what travels with gaze_target, using the names and rates the episode schema and capture specs state. The gaze row itself is deliberately thin: the schema names the field, and its data type, shape and frame are not itemised on the site.

Because every field sits on one timeline, a gaze annotation can be read against the head camera at 60 fps, the 200 Hz head IMU and the hand joints for the same moment.

Gaze is not head motion. The head IMU records how the head moved at 200 Hz, 6-DoF; gaze_target records where attention was directed. Gaze is also not hand pose, which is the per-joint 3D position of the fingers, and it is not an action segment, which is a verb and noun with start and end times.

These combine in practice. The differentiation table shows what each covers so you can specify exactly the fields you need.

Every sensor runs in one PTP clock domain and exposures are hardware-triggered. The site states a 1.12 ms median offset, 1.94 ms at the 99th percentile and a 2.0 ms ceiling, measured as the absolute timestamp offset between the head camera and every other stream.

The streams run at different rates: cameras at 60 fps, depth at 30 fps and the IMUs at 200 Hz. The site does not state the rate at which gaze_target is sampled, so how it is aligned to a video frame is set per program in the skill spec.

All formats are exported from the same QA-passed episodes as projections of the canonical episode schema. If you plan to read gaze_target directly, ask for HDF5 or zarr, whose annotations/ group mirrors the episode schema.

For RLDS, LeRobot and WebDataset the site states how hand joints and actions are carried but not a gaze key, and says which fields appear inside a shard’s ann.json is set in the skill spec. Where gaze sits in those layouts is therefore set per program.

Paid, consented contributors perform the real task against your spec while wearing the rig. We do not capture robot or teleoperation data, and we do not capture at a customer’s own facility. Every episode is tied to a signed contributor release, and faces, plates and screens in frame are detected and irreversibly blurred at ingest, with a reviewer confirming the result.

The dataset ships under a perpetual, irrevocable, worldwide commercial licence that is non-exclusive by default, with the buyer owning the resulting dataset and exclusivity available. Where coverage is deep, the median time from signed spec to first validated batch is a median of 48 hours; where it is thin, turnaround runs three to four weeks. Billing is per validated hour, not per raw hour recorded, and there is no list price: you are quoted against your spec.

Write down the tasks, environments and objects your model needs, say whether gaze is in scope and how you expect to read it, and name the delivery format. Most orders begin with a Spec Pilot: a small, fast first batch that proves the spec and accept criteria hold on real footage before you scale.