How Much Annotation Does Robot Training Data Need?

The honest answer is that "how much" is the wrong first question. Annotation density is not a fixed percentage or a labels-per-hour rate that applies across every project — it is a decision that follows from what a policy needs to learn and at what stage of training it needs to learn it. Two episodes of the same raw egocentric video can ship with very different label sets depending on whether they are headed into broad pretraining or a narrow post-training run for one skill.

What is consistent, per /formats/episode-schema, is the vocabulary of labels available to apply: hand pose keypoints with visibility flags, action segments with verb/noun and start/end timestamps, contact events, and instance segmentation masks, all tied to synchronized video, depth, and IMU rather than to an isolated RGB frame. The question that actually has an answer is which of those label types a given batch needs, at what granularity, and across how much of the footage — and that is exactly what a skill spec exists to pin down before collection starts.

For the mechanics of how those labels get attached to footage — calibration, capture, and QA — see /guides/how-robot-pretraining-data-is-annotated. For the distinction between segment-level and per-frame labels referenced throughout this page, see /guides/skill-level-vs-frame-level-annotations.

Pretraining data generally needs annotation that is broad rather than exhaustive: action segments and hand pose across a wide range of tasks, environments, and objects, so a foundation model can learn general manipulation priors rather than one narrow behavior. The density that matters at this stage is coverage across diversity — many different skills and settings each labeled with the same core fields — rather than dense per-frame labeling on any single clip.

Public pretraining research reaches similar label sets through a different route. EgoScale (arxiv.org/abs/2602.16710) derives its hand-pose and camera-trajectory labels as pseudo-labels from SLAM and hand-pose estimation models run over egocentric footage at scale, rather than from human annotators measuring each frame. GR00T N1 (arxiv.org/abs/2503.14734) similarly relies on latent actions and an inverse-dynamics model to produce action labels it did not collect by hand. Both are evidence that pretraining-scale corpora lean on automated or semi-automated labeling to reach broad coverage, which lines up with why pretraining specs on Firsthand call for the same core fields applied consistently across many episodes rather than maximal density on a small set.

As training narrows from broad pretraining toward a specific skill or robot, the same label vocabulary gets applied more precisely to a smaller, more targeted set of episodes rather than expanding into new label types. Mid-training and post-training batches for one manipulation skill typically call for tighter contact-event timing, more consistent verb/noun labeling for the exact sub-actions that skill involves, and hand-pose coverage on the specific grasps that skill requires — applied to episodes chosen for that skill, not spread across unrelated tasks.

This is a shift in precision and selection, not a shift in the tools available. The same annotation fields documented at /formats/episode-schema and offered through /custom-collection/egocentric-video-annotation apply at every stage; what changes is how tightly a skill spec scopes which episodes get annotated and how carefully QA checks the labels that matter most for that one skill.

Which label types carry the most weight also depends on the use case, not just the training stage.

For manipulation skills, hand pose and contact events tend to matter most, since grasp shape and the moment of contact are what a policy needs to imitate. For humanoid and whole-body tasks, action segments that capture longer, multi-step sequences matter alongside hand pose, since the skill spans more than a single grasp. For world-model or representation-learning use cases, broad action-segment and object-mask coverage across diverse scenes tends to matter more than fine-grained hand pose on any single episode, since the goal is general scene and dynamics understanding rather than one manipulation behavior.

A handful of factors push annotation density up or down for a given project, and a skill spec is where they get weighed against each other:

• How many distinct sub-actions a skill involves — more discrete steps means more action-segment boundaries to label per episode.
• Whether object identity matters to the policy — if it does, instance segmentation masks get added; if the task is purely about hand and arm trajectory, masks may not be needed at all.
• How tightly the target skill is scoped — a broad pretraining batch spreads the same label fields across more episodes and more diversity; a narrow post-training batch concentrates labeling effort on fewer, more specific episodes.
• How much the downstream model already handles through pseudo-labeling or self-supervision — where a training approach (like GR00T N1’s inverse-dynamics model) can derive some signal from unlabeled data, the human-labeled portion of a batch can be scoped down accordingly.

These are directional factors for scoping a spec, not a formula. The specific density for a project is set in the skill spec and checked on a pilot batch, described below.

Pseudo-labels can extend coverage, but they are a different source of signal than measured annotation, and the two are not simple substitutes for each other. EgoScale’s hand-pose and trajectory labels come from SLAM and hand-pose estimation models run over RGB video after the fact; GR00T N1’s action labels come from latent actions and an inverse-dynamics model rather than from a human or a calibrated sensor rig observing the action as it happened. Both approaches are well suited to reaching large volumes of data cheaply, which is exactly why they show up in large public pretraining corpora.

Firsthand’s annotation instead ties labels to synchronized video, depth, and IMU captured on a calibrated rig at the time of recording, as described on /custom-collection/egocentric-video-annotation and /guides/how-robot-pretraining-data-is-annotated. That measured approach is the right choice when a project needs labels it can trust at the precision level mid-training or post-training work depends on. Whether pseudo-labels are an adequate substitute for a given project is a training-methodology question that depends on what the downstream model architecture can tolerate — which is one more reason density and label sourcing get decided in a spec rather than assumed from a template.

A skill spec is where a buyer and Firsthand agree on which label fields a batch needs, at what granularity, and across how many episodes — before a full collection run starts. Rather than guessing at density in the abstract, the spec states the label types (from the fields in /formats/episode-schema) the downstream model needs and the scope of footage they apply to.

That spec is then checked against a small pilot batch before scaling to the full collection. The pilot surfaces whether the specified label density actually produces usable training data for the intended model, and lets the spec be adjusted — more or less coverage, additional or fewer label fields — before committing to the full run. This pilot-then-scale sequence is the same pattern described for collection scope generally in Firsthand’s custom collection process, applied here specifically to annotation.

Whatever density a spec calls for, the labels that get delivered go through the same quality verification step regardless of how much or how little a given batch needs. That QA step checks that action-segment boundaries, hand-pose keypoints, and contact events are consistent and correctly tied to the synchronized streams they describe, rather than spot-checking a sample after delivery. Data-labeling quality verification is covered in full at /custom-collection/data-labeling-quality-verification.

This matters for density decisions specifically because QA cost scales with the label fields a spec asks for, not just with episode count — a spec that adds instance masks on top of hand pose and action segments is asking for more verification work per episode, which is one more input into scoping a spec correctly the first time rather than over-specifying every field by default.

Whatever density a spec settles on, it shows up in the delivered episode as a specific subset of the annotation fields already documented at /formats/episode-schema — not as a separate density metric attached to the file.

A lower-density pretraining batch might deliver only hand pose and action segments across many episodes; a tightly scoped post-training batch might add instance masks and tighter contact-event timing on a smaller episode set. Either way, the fields themselves come from the same schema — density is expressed in which fields are populated and across how much footage, not in a different file format.