What Is a Vision-Language Model (VLM)?

A VLM is only as good as the pairing it learns from. Footage with no true language, or language that describes something other than the footage, teaches the wrong relationship. The language side is where datasets differ most, so it is the part to specify and verify before collection starts.

Firsthand captures custom first-person video with one task instruction paired to each episode on the same clock as the video, plus action segments, hand pose and synchronized streams, collected fast, on demand, and to a written spec. Captions and narration are not stated as delivered, and delivery ships under a buyer-owned license.

A vision-language model, or VLM, is a model that takes images and text together and learns how the two relate. In the GR00T N1 paper (arXiv 2503.14734) the VLM backbone pairs an image encoder with a language model: each image is encoded into tokens, and those tokens are then processed together with the text by the language model. The paper describes the backbone as pretrained on Internet-scale data and aligned over a broad set of vision-language tasks.

The term is often used loosely. A VLM is not a robot policy, and it does not produce an action by itself. It interprets what is in view and what is being asked. In the paper that is the job of the System 2 module, which processes the robot’s visual perception and language instruction to interpret the environment and understand the task goal. Searches for VLM egocentric video usually mean first-person footage paired with language, which is what the rest of this page covers.

A VLM relates images to language. A vision-language-action (VLA) model also produces an action. The paper describes GR00T N1 as a VLA model that generates actions from image and language instruction input, and it splits the work in two. A pre-trained VLM is the System 2 reasoning module, running at 10 Hz. A diffusion transformer is the System 1 action module, and it cross-attends to the VLM output tokens. The paper reports 1.34B of the released model’s 2.2B parameters in the VLM.

So a VLM can be a component inside a VLA model, or a model in its own right. The data consequence follows. A VLM needs frames paired with true language. A VLA model needs that pairing plus the action stream that carried the task out, all on one synchronized clock, as the vision-language-action model entry explains. If the goal is a VLM, the action fields are optional extras. If the goal is a VLA model, language-only data is missing a leg. The first table below sets the two side by side.

A VLM learns from pairs: a frame or clip, and language that says something true about it. The paper describes its backbone as pretrained on Internet-scale data before any robot training, and in policy training it passes a text description of the task, together with one or more images, to the VLM. The paper treats its human video datasets as the base of its data pyramid and describes them as providing detailed first-person perspectives of hand-object interactions.

What the language contains is where datasets differ. It might be an instruction the person was following, a label for a step, or a description of what happened. For Firsthand capture, the one language field the site states is a task instruction paired to each episode. Captions, narration and dense descriptions are not stated as delivered, so this page does not present them as part of a collection.

A first-person recording keeps the hands, the object they are touching and the action in progress in view together, continuously, rather than small in a wide shot from across a room. For a model that has to connect a sentence to something visible, that is a cleaner thing to connect it to. Hand pose is delivered at 21 joints per hand in 3D with a per-joint visibility flag, so a hidden finger is flagged rather than guessed.

Custom capture also means the content follows your task list. Public footage was recorded for someone else’s purpose, so its scenes, tasks and objects are whatever its publisher chose. A custom collection starts from a written skill spec naming the tasks, settings, objects and labels you need. Where the gap in a corpus is a particular kind of task or setting, that is a more direct way to close it than searching for footage that happens to match.

A VLM-oriented egocentric collection should settle five things in the spec. First, language: the task instruction, paired to the episode on the same synchronized clock as the video, with wording either a fixed instruction set or freeform. Second, step structure: action segments, each with start and end timestamps, a verb and a noun. They give temporal structure and a small consistent vocabulary, but they are not free-form captions.

Third, the visual signal: ten synchronized streams including a 4K head camera, two wrist cameras and depth recovered from the calibrated stereo rig, hardware-triggered onto one PTP clock and aligned under 2 ms. Which streams an order carries is decided in the spec. Fourth, rights: each contributor signs a documented release, and faces, plates and legible screens are anonymized. That also means text on a screen is not in the delivered pixels, which matters if your model must read text in the scene.

Fifth, delivery: RLDS, LeRobot dataset format, WebDataset, or HDF5 and zarr, with the format set before capture begins. Calibration is accepted only under a 0.28 px RMS reprojection gate, and episodes that fail checks are rejected rather than delivered with a caveat. The second table below lists each language item and its status as the site states it.

Write a skill spec: the tasks and their ordered sub-steps, the settings and objects, the streams and labels, the accept criteria and the delivery format. Then add the language decisions the site leaves open: whether the instruction is a fixed set or freeform, whether you need sentence-level step descriptions, and whether transcripts or audio are in scope. Many programs begin with a Spec Pilot, a small first batch captured against the spec and reviewed with you before the full order runs.

On speed, a first validated batch arrives in a median of 48 hours from signed spec where coverage is deep, and in three to four weeks where contributors have to be recruited first. That describes the capture pipeline, not how long a language-annotation scheme takes, which the site does not state. Put each unstated item in the request as a requirement. If you need sentence-level descriptions, say so and ask what the pilot would show.

The license is perpetual, non-exclusive and buyer-owned, for commercial training use. The site does not claim that any model has been trained on Firsthand data, and it states no hour, episode or contributor counts.

What is a vision-language model (VLM)?

A VLM takes images and text together and learns how they relate. In the GR00T N1 paper the backbone pairs an image encoder with a language model, so the model can interpret a scene and a written instruction. A VLM does not produce an action by itself.

Is a VLM the same as a VLA model?

No. A VLA model also outputs an action. In GR00T N1 a pre-trained VLM is the System 2 reasoning module, and a separate diffusion transformer is the System 1 action module that cross-attends to the VLM output tokens.

What training data does a VLM need?

Frames or clips paired with language that is true about them: an instruction the person was following, a label for a step, or a description of what happened. Footage without true language, or language that does not match the footage, teaches the wrong relationship.

Why use egocentric video for a VLM?

A first-person view keeps the hands, the object and the action in progress in frame together, which is a cleaner thing for a sentence to refer to. Custom capture also lets the tasks, settings and objects follow your written spec instead of whatever a public corpus happens to contain.

Does Firsthand deliver captions or narration for VLM training?

Not as stated. The language field the site states is one task instruction paired to each episode, plus action segments with a verb and a noun. Transcripts are an annotation option chosen in the spec, with format and coverage not stated. Narration and dense captions are not stated as delivered.

How fast can a custom VLM dataset be delivered?

A first validated batch arrives in a median of 48 hours from signed spec where coverage is deep, and in three to four weeks where contributors must be recruited first. The site does not state how long a language-annotation scheme takes.

What formats and license come with it?

Delivery is in RLDS, LeRobot dataset format, WebDataset, or HDF5 and zarr, with the format set in the spec before capture. The license is perpetual, non-exclusive and buyer-owned, for commercial training use.