What Is a Language-Conditioned Robot Policy?

A language-conditioned policy lets one model do different tasks depending on the instruction, instead of needing one policy per task. The RT-2 paper describes training vision-language models to act as instruction-following robotic policies, and the Octo paper describes Octo as a policy that can be instructed with language commands or goal images. Put plainly, the instruction is one more input next to the camera images.

Firsthand does not train, ship or benchmark language-conditioned policies, and it captures first-person human video, not robot or teleoperation data. It delivers the fields the site states: head and wrist camera video, 21-keypoint 3D hand pose per hand with a per-joint visibility flag, wrist trajectories, action segments, contact events, 200 Hz IMU and calibrated stereo depth, with one task instruction per episode. That is the only language field. Firsthand is not affiliated with the authors of any paper cited here and makes no claim that any named model was trained or evaluated on its data.

At each step the policy takes observations, such as camera images, together with the instruction, and outputs an action or a chunk of actions. The instruction is an input, not an output: change it and the same scene can yield different behavior.

The mechanics differ by model. RT-2 writes the task description into a text prompt and has the model output action tokens, while Octo tokenizes the instruction with a pretrained language model and processes it with image tokens in a transformer. The first table sets out each source’s description.

RT-2 takes a robot camera image and a textual task description and outputs a string of numbers or least-frequently-used tokens that represent a robot action. Its robot data is annotated with a natural-language instruction made of a verb for the skill and one or more nouns for the objects. The OpenVLA paper states that, given an image observation and a language instruction, the model predicts robot control actions, and that it fine-tunes a pretrained Prismatic VLM backbone.

The Octo paper describes input tokenizers for language instructions, goals and observation sequences, with language passed through a pretrained T5-base transformer and a diffusion head producing chunked actions. The π0 paper says its observation consists of multiple RGB images, a language command and the robot’s proprioceptive state, with an action expert using flow matching. The GR00T N1 paper says its vision-language module interprets vision and language instructions and a diffusion transformer generates motor actions.

The EgoScale paper describes a flow-based architecture, similar to GR00T N1, that conditions on an image and a language instruction and predicts a chunk of future actions. Details these sources do not give are marked Not stated in the sources read.

An unconditioned policy takes no task input and produces one behavior from observations alone. A goal-conditioned policy is told the target, often as a goal image. A language-conditioned policy is told the task in words. These are plain-word distinctions, not definitions from any one paper, and they can combine.

Octo shows the combination. Its paper says Octo can be conditioned on language instructions or goal images, that training randomly zeroes out the language instruction or goal image per example, and that datasets without language annotations always use goal-image conditioning, which reduces the burden on language annotation.

Each training episode needs three things paired on one timeline: the observations, the action signal that carried out the task, and the instruction that names it. The vision-language-action model entry covers this triple. RT-2 trains on robot trajectories annotated with an instruction, and Octo’s recipe shows that language can be absent for some data when goal images fill in. How many episodes or how varied the wording must be is Not stated in the sources read.

Instruction granularity also varies. The π0 paper evaluates tasks that require following a sequence of intermediate language commands, such as the subtask of picking up a napkin, and it describes pre-training language labels that combine task names with segment annotations for sub-trajectories. A single instruction per episode is a coarser label than that. Whether a given policy needs finer language than one instruction per episode depends on the model and the task, and the sources read do not state a general rule.

Language is only one of the conditioning inputs. Octo adds goal images, π0 adds proprioceptive state and GR00T N1 adds robot state, so a dataset spec should list every input the target model consumes, not just the instruction. Firsthand human capture covers the human-side observation and hand-action fields in the second table. Robot state and robot actions are not part of it.

You need it when your tasks, environments or hand signals are not covered by data that already carries an instruction. Firsthand collects fast, on demand and to a written spec, billed per validated hour, so the spec is where you settle the fields, the format and the exact wording of the single task instruction per episode. The site states no mapping from its fields to any robot’s action space or any model’s input format, so that mapping is Not stated and stays with your pipeline.

What is a language-conditioned robot policy?

It is a policy whose actions depend on a natural-language instruction as well as observations such as camera images, so the same scene can produce different behavior for different instructions.

Is a language-conditioned policy the same as a VLA model?

No. The term describes how a policy takes its task. A VLA is a model type that takes images and a language instruction and outputs actions. The Octo paper calls Octo a generalist robot policy.

Can a policy use a goal image instead of language?

The Octo paper says Octo supports both language instructions and goal images, and that datasets without language annotations always use goal-image conditioning.

Does Firsthand provide per-step or dense language labels?

No. The only language field the site states is one task instruction per episode, on the same clock as the video and action streams.

Does Firsthand deliver robot or teleoperation data?

No. Firsthand captures first-person human video. Mapping it to your robot is a step in your own pipeline.

Was any named model trained on Firsthand data?

Firsthand makes no such claim and is not affiliated with the authors of the papers cited here.

Can I choose the wording of the task instruction?

Request a custom dataset and write the instruction wording, fields and format into the spec.