Instruction-Labeled Egocentric Task Data

Instruction-labeled egocentric task data is first-person video of a person doing a task, where each recorded episode is paired with one task instruction that names what that episode is for. The camera sits on the contributor’s head, so the frames show the task from roughly the point of view a robot would have. Alongside the video the rig records synchronized sensor streams, and the instruction travels with them as one field of the episode.

The idea behind the pairing is simple. A language-conditioned policy is a policy whose action output depends on an instruction as well as on what it observes, and it is commonly trained by imitation from demonstrations in which each episode is paired with the instruction that describes the task. Video and action alone tell a model what happened. The instruction tells it what it was being asked to do, which is the part a language-conditioned policy conditions on.

This page is deliberately narrow about language. The site states one task instruction per episode and no other language field for Firsthand capture. It states no narration track, no captions, no dense language annotation, no image-text pairs, no question answering and no step-by-step instruction text. Where a page below says "Not stated", that is the whole answer: the site does not say.

Firsthand captures first-person human video from paid, consented contributors. It is not robot data and not teleoperation data, and every field below is human capture. On the instrumented tier an episode carries ten hardware-synchronized streams: a 4K head camera, two wrist cameras, depth, IMU, 3D hand and body pose, instance segmentation and action labels. The streams are hardware-triggered onto one PTP clock domain and aligned under 2 ms.

The action fields are the ones the site already states. Hand pose is 21 joints per hand in 3D, for both hands, with a per-joint visibility flag, so a joint hidden behind an object is flagged rather than guessed. Wrist trajectories come from the same capture. Action segments carry a start time, an end time, a verb and a noun per step as written in the skill spec. Contact events mark when a hand meets an item. IMU is 6-DoF at 200 Hz from head- and wrist-mounted sensors, and metric depth is recovered from the calibrated stereo rig.

Then comes the instruction. One task instruction is delivered with each episode, on the same clock as the rest. The site does not state the exact name of that field, how long it is, or what language it is written in, so the table below marks those as "Not stated" rather than inventing a schema.

Which of the ten streams a phone-captured episode carries is not stated, so if your plan depends on a particular stream, say so in the spec and confirm the capture setup for that task before you commit.

The site states two things about the instruction. There is one per episode, and it is paired to the episode on the same synchronized clock as the video and the action stream. That is the full extent of what it states about attachment, and this page does not go further than that.

It does not state who writes the instruction, whether it is free text or drawn from a fixed template, whether it is written before or after the contributor performs the task, how it is reviewed, or whether several phrasings of the same task are supplied. All of those are "Not stated". The site also states no paraphrase set, so one instruction per episode should be read literally: a single instruction, not a family of rewordings.

What the site does give you is the place to settle those questions: the written skill spec. The spec names the tasks, the verb and the noun for each step, and where each step starts and ends. If your model expects a particular instruction style, put it in the spec as a request and ask for it to be scoped, since it is a decision between you and the team collecting the data rather than a stated default.

Because the action segments use a verb and a noun per step, it is worth deciding early whether you want your instruction wording to be consistent with that vocabulary. The site does not say the two are linked, so treat any such consistency as a spec decision, not a built-in behavior.

Published robot policies take language as an input next to the camera images, and the papers describe it in different ways. The RT-2 paper describes a model that takes a robot camera image and a textual task description and outputs a robot action, and says its robot data is annotated with a natural-language instruction made of a verb for the skill and one or more nouns for the objects. The OpenVLA paper states that, given an image observation and a language instruction, the model predicts robot control actions.

The π0 paper says its observation consists of multiple RGB images, a language command and the robot’s proprioceptive state. The GR00T N1 paper says its vision-language module interprets vision and language instructions, and that a diffusion transformer generates motor actions. The common thread is only what the papers state: an instruction is an input, and a policy that conditions on it needs examples in which the instruction is paired with the behavior.

That is the narrow reason an instruction per episode matters. If every episode is paired with the instruction that describes its task, a team can use the dataset where a model is conditioned on language, and can use the same episodes without the instruction where it is not. The site makes no claim about which model needs what, and no claim about results.

It is worth being plain about the limits. The papers above describe robot data and their own pipelines. Firsthand delivers human first-person data and is not affiliated with the authors of any of them. The site makes no claim that RT-2, OpenVLA, π0, GR00T N1 or any other named model was trained, fine-tuned or evaluated on Firsthand data, and bridging human data to a particular robot is a step your own pipeline performs after delivery.

The site states one task instruction per episode as the only language field tied to Firsthand capture. Everything else in the language family is therefore not part of the stated deliverable. That includes spoken narration recorded during the task, captions on frames or segments, dense language annotation, image-text pairs, question-and-answer pairs, subtask language, step-by-step instruction text, paraphrases and reasoning traces.

This is a boundary, not a gap in the page. A model that needs a different language layer should know up front that this dataset does not state one, and the second table below sets out each item with its status. If you need more than one instruction per episode, treat it as a custom request that has to be scoped and priced against a written spec, and do not assume it is included.

The site also states no head or camera pose, no map and no gaze or eye-tracking stream as a deliverable. The only IMU the site states is the 200 Hz 6-DoF one. Keeping those out of this page matters because instruction-labeled data is sometimes assumed to come with a richer set of labels than it does.

Every episode is machine-checked for sync and calibration drift, then reviewed by a human against the task spec. Calibration is run twice per session and accepted only under the 0.28 px RMS reprojection gate. Rejected episodes are recaptured at the provider’s cost, and a reject log ships with every batch, so a buyer can see what did not make the cut and why.

Privacy is handled at ingest. Faces, plates and screens are irreversibly blurred, a human reviewer confirms the result, and an anonymization record is delivered. If a reviewer is unsure a region is fully covered, the episode is held or discarded. Screen content is not a deliverable, and nothing on this page describes capturing identifiable people.

Episodes ship in RLDS, LeRobot, WebDataset or HDF5, with per-stream nanosecond timestamps, and the buyer owns the delivered data under a perpetual license. The LeRobot repository describes its dataset format as synchronized MP4 videos or images plus Parquet files for state and action data, the RLDS repository describes RLDS as an ecosystem of tools to store, retrieve and manipulate episodic data, and the WebDataset repository describes a tar-shard format in which files sharing a basename form one training sample.

Where the instruction lands inside each of those formats is not stated, so ask for a sample before you build a loader. The sample data outputs page shows the delivered files for a recording.

The site states a median of 48 hours from a signed spec to a first validated batch where coverage for your tasks and regions is deep, and three to four weeks where coverage is thin. It does not say which of those applies to a given instruction-labeled request, and it states no list price. Billing is per validated hour, and rejected items are recaptured.

The site does not state durations, episode counts, collection sizes or throughput for this topic, so none are given here. Those numbers are set in the written skill spec, which is where reach, timeline and price are scoped before any capture begins.

In practice the fastest route is a short, concrete spec. State the tasks, the setting and the variation or failure cases you want to see, and say how you want the instruction worded. The more of that is decided up front, the less there is to scope, and the first batch can be judged on a spec that was clear from the start.

Start with the tasks. Pick a small set of tasks your model is expected to follow an instruction for, and write each as a verb and a noun with a clear start and end. That mirrors how the site describes action segments, and it gives the instruction something precise to describe.

Then write down what you need from the instruction that the site does not state: its wording, its language, whether you want a fixed template, and whether one phrasing per episode is enough for your model. Those are scoping decisions for the spec, and flagging them early avoids a surprise at delivery.

Finally, ask for a sample and a quote. The team scopes reach, timeline and price against the written spec, and a first validated batch lets you confirm the instruction field, the format you want and the loader you will use before a larger collection. Request a custom dataset below with the tasks, the setting and the format.