Egocentric Video Data for Vision-Language Model Training

Egocentric video is video recorded from the wearer’s own point of view, usually from a head-mounted camera with wrist cameras alongside. For a vision-language model, the video is only half of a training example. The other half is language that says something true about the footage: an instruction the person was following, a label for a step, or a description of what happened. Egocentric video data for vision-language model training is that pairing, captured on purpose and delivered in a form a training pipeline can load.

This page is deliberately narrow about the language half. The site states one language field for Firsthand capture: a task instruction paired to each episode. It states step-level labels in the form of action segments, each with a verb and a noun. It lists transcripts as an annotation option chosen in the spec, without stating a format or coverage. It does not state narration or dense captions as delivered. Each of those statements is repeated, with its status, in the table further down.

It sits next to three pages that already exist without repeating any of them. The multimodal page covers hands, objects and actions for video models in general. The vision-language-action pretraining page covers broad pretraining for action models. The annotation page covers the label layer. This page asks a different question: what does a team training a vision-language model get on the language side, and what should it ask for in the spec?

A first-person recording keeps the manipulation as the dominant content of every frame. The hands, the object they are touching and the action in progress are in view together, continuously, rather than appearing small in a wide shot from across a room. For a model that has to connect a sentence to something visible, that is a cleaner thing to connect it to.

Custom capture also means the content follows your task list. Public footage was recorded for someone else’s purpose, so its scenes, tasks and objects are whatever its publisher chose. A custom collection starts from a written skill spec: the tasks, the settings, the objects and the labels you need. Where the gap in a corpus is a particular kind of task or setting, that is a more direct way to close it than searching for footage that happens to match.

Speed is stated on the site and is the same here as for any collection: a first validated batch arrives in a median of 48 hours from signed spec where coverage is deep, and in three to four weeks where contributors have to be recruited first. That is a statement about the capture pipeline, not about how long a particular language-annotation scheme takes, which the site does not state.

One is stated without qualification: a task instruction paired to each episode. The vision-language-action pretraining page describes it as attached on the same synchronized clock as the video and action stream, so the language, the vision and the action are simultaneous within one episode. That page also says the instruction wording can be written to a fixed instruction set or left freeform, and that this is decided in the spec.

Step-level structure is stated through action segments. Each segment carries a start and end timestamp, a verb and a noun. That is structured text of a sort, a short verb-noun pair for each step, but it is not a free-form caption and this page does not present it as one. A team that wants sentence-level descriptions of each step should say so in the request.

Transcripts are listed as an annotation option on several pages, described as labels for what is said, chosen in the spec. The egocentric audio page is more cautious: it lists microphone, channels, sample rate, codec, audio-to-video offset and transcripts as set per program in the skill spec, and its own comparison marks transcripts for egocentric episodes as not stated. The honest reading is that transcripts can be scoped, not that a defined transcript product exists. Narration and dense captions are not stated as delivered, and other pages on this site say so directly.

Video shows what happened. It does not, on its own, say what the person was asked to do. An instruction attached at the episode level supplies that. It lets a model learn the relationship between a request in words and the behavior that satisfied it, which is the relationship an instruction-following model conditions on. The vision-language-action page makes the same point from the action side: without an instruction at the episode or step level, a video-and-action pair shows what happened but not what the model was being asked to do.

Because the instruction is paired per episode, one episode is one instruction and one attempt. That is a clean unit for a training example, and it is easy to audit: you can read the instruction, watch the attempt and judge whether the two match. It also means the instruction describes the whole episode. It is not a running commentary on each second, and the site does not state one.

The wording matters more than it first seems. A fixed instruction set gives a model many examples of the same phrasing and makes evaluation tidy. Freeform wording gives it varied phrasing but is harder to hold consistent. Neither is better in general, and the site leaves the choice to the spec. If your model will be prompted by end users in their own words, say so, because it shapes which approach to ask for.

An action segment divides a task into start- and end-timestamped spans, each carrying a verb and a noun. A task such as sign and file a form becomes an ordered run of labeled steps rather than one long clip with one label. The timestamps are on the same clock as every other stream, so each step lines up with the video frames, the depth, the IMU and the hand pose for that span.

For a vision-language team this gives two things short of a caption. It gives a temporal structure: where each step starts and ends. It also gives a small, consistent vocabulary: the verb and noun for each step. Those can be used as supervision directly, or as the skeleton on which a team builds its own descriptions downstream. Whether to expand them into sentences is your pipeline’s decision, and the site does not state that Firsthand does it.

Contact events add another structured signal where hand-object contact matters: they mark when a hand meets or releases an item. Hand pose is delivered at 21 joints per hand in 3D with a per-joint visibility flag. None of these is language, but each ties a stretch of video to something a sentence might refer to, which is useful when a team is aligning its own text to the footage.

The Ego4D paper describes dense, timestamped narrations of camera-wearer activity written by annotators, along with natural language queries and speech transcriptions, over a large body of unscripted daily-life video from many camera wearers. That is a large, published research resource, and its documentation is the place to read what it contains and under what license. Firsthand has no relationship with its authors, and nothing here is a claim about its quality.

The difference that matters for buying is the shape of the data. A public corpus gives you what its publisher recorded and annotated, in its own scenes and tasks. A custom collection gives you the tasks you name, in the settings you name, with the streams and labels you specify. On the language side the trade is plain: a public narrated corpus offers dense text that the site does not state Firsthand delivers, and a custom collection offers a per-episode instruction paired to footage you commissioned, under a license you own.

Many teams will use both: a public corpus for breadth and early experiments, and a custom collection to fill a specific gap in tasks, objects or settings. The comparison table below sets the two side by side, with the public column limited to what the paper itself states.

The stream set is the same as for any custom egocentric collection: ten synchronized streams including a 4K head camera, two wrist cameras, depth, IMU, 3D hand and body pose, instance segmentation and action labels. Depth is recovered from the calibrated stereo rig. The streams are hardware-triggered onto one PTP clock domain and aligned under 2 ms, the IMU is 6-DoF at 200 Hz, and every stream carries per-stream nanosecond timestamps.

Which of those a given order carries is decided in the skill spec. A team that only needs video and an instruction can ask for exactly that; a team that wants depth and hand pose for a richer signal can ask for them. The site states the full stream list, but not a per-order default, so the spec is where it is settled.

The table below lists each stream and label with the detail the site states, and says what a vision-language team might use it for. The use column is a suggestion, not a claim that any model was trained on the data.

Write it as a skill spec: the tasks to demonstrate, the settings and objects, which streams and labels the pipeline needs, the accept criteria and the delivery format. For a vision-language team, add the language decisions that the other pages leave open: whether the instruction is a fixed set or freeform, whether you need sentence-level step descriptions, whether transcripts matter, and whether audio is in scope at all.

Decompose the tasks into ordered sub-steps. Each becomes a labeled action segment with a verb and a noun, so the spec controls the granularity of the step labels. Name the objects too, because the noun in each segment comes from the vocabulary the spec establishes. If a task is prone to going wrong, ask for failure and recovery cases deliberately, so the language and the footage cover them.

Many programs begin with a Spec Pilot: a small first batch captured against the written spec, reviewed with you before the full order runs, so that instruction wording, label quality and coverage can be corrected before they are repeated at volume. The flow the site states is spec, Spec Pilot, then scale. Price and quote terms are not stated on this page.

Every contributor signs a documented release as part of a consent chain, and the record travels with the data through delivery. A vision model runs on every frame at ingest, flagging faces, heads, licence plates and any legible screen or document, and those are irreversibly blurred. A human reviewer confirms the result before delivery, and clips where a bystander cannot be de-identified are discarded rather than shipped with a caveat.

Language adds a point worth stating. If audio is in scope, the audio page says it is a spec decision rather than a default, that speech is scrubbed of names, addresses and card numbers, and that ambient audio is retained. That matters for a team that wants spoken language in the data, because the scrubbing applies to what is said as well as to what is seen. Whether a given engagement can keep specific spoken content is not stated, so raise it before the spec is signed.

The same applies to written text in frame. Legible screen and document content is blurred at ingest, so it is not in the delivered pixels. A model that must read text in the scene is not served by footage in which that text has been removed, which is worth knowing before you ask for language grounded in what a screen or page says.

Calibration runs twice per session and is accepted only under a 0.28 px RMS reprojection gate. Every episode passes calibration and synchronization checks on every active stream, plus label and annotation review for hand pose and action segments where those are included. Episodes that fail are rejected rather than delivered with a caveat, and the reject log is delivered with a Spec Pilot batch so you can see what was refused and why.

Delivery is in RLDS, LeRobot dataset format, WebDataset, or HDF5 and zarr, whichever your pipeline already reads, with the format set in the spec before capture begins. The task instruction and the other fields are carried as structured fields. A field path for the instruction is not stated on the pages checked for this one, so confirm it with the data card at the Spec Pilot.

The license is perpetual, non-exclusive and buyer-owned, for commercial training use, with consent, anonymization and provenance documentation that your legal team can review. Exclusivity is a separate question that the site treats on its own page.

Several things a vision-language team might assume are not stated on this site, and this page does not supply them. There is no narration or dense-caption product, no caption-length or caption-quality figure, no inter-annotator agreement figure and no label accuracy figure. Languages other than the instruction’s own, and translation, are not stated. The format and coverage of transcripts are not stated. No train and evaluation split is stated.

Counts are not stated either: not the number of contributors, not the number of episodes, not the hours that exist today. This page claims no hours collected, no throughput and no customer, and it does not claim that any model has been trained on Firsthand data.

The practical advice is to put each of these in the request as a requirement rather than an assumption. If you need sentence-level descriptions of every step, say so, and ask what the Spec Pilot would show. If a requirement turns out not to be offered, you will learn that at the pilot, before volume, which is the point of running one.