What Are Pseudo-Labels in Robot Learning?

Most egocentric human video and web-scale footage was never recorded with a robot present, so it carries no ground-truth action label — no joint angle, gripper command, or wrist trajectory. Pseudo-labeling lets that unlabeled footage still contribute action-relevant signal: a model infers what action most plausibly explains an observed change, or learns a compressed representation of the change itself, and that inferred label stands in for the ground truth that was never captured. The tradeoff is that a pseudo-label is only as good as the model or heuristic that produced it, and it carries no independent verification the way a human-checked or directly-measured label does.

Firsthand does not produce action signal by pseudo-labeling. Egocentric video, depth, 3D hand pose, wrist trajectories, contact events, and discrete action segments are captured and annotated directly at the time of recording, on calibrated and synchronized capture hardware, then checked against the written spec in QA before delivery in RLDS, LeRobot format, or WebDataset with a buyer-owned commercial license. That gives a buyer a label that was measured or annotated against footage, not inferred by a state-transition model or a learned latent representation after the fact — though the two approaches are not mutually exclusive, and a directly-labeled dataset can still be blended with a wider pool of pseudo-labeled video.

What is a pseudo-label?

A pseudo-label is a label a model produces for a piece of data, rather than a label recorded directly at capture time or verified by a human against a spec. In robot learning it most often stands in for action ground truth on footage that has none.

How are pseudo-labels generated for robot data?

By running a model over unlabeled footage after the fact. GR00T N1 uses an inverse dynamics model to infer the action behind an observed state change, and separately learns latent actions — a compressed representation of what changed between frames — directly from video, with no explicit label ever recorded.

How does GR00T N1 use inverse-dynamics pseudo-labels and latent actions?

NVIDIA’s GR00T N1 paper (arXiv 2503.14734) applies inverse-dynamics pseudo-labeling and latent actions at the wide base layer of its data pyramid, letting large volumes of unlabeled egocentric human video still contribute action-relevant training signal even though no robot was present when it was recorded.

How does EgoScale derive action labels from human video?

EgoScale (arXiv 2602.16710) derives a wrist-level action representation from egocentric human video using off-the-shelf SLAM and hand-pose estimation, rather than a human annotator labeling each clip or a robot being present at capture.

What is the difference between a pseudo-label and a label from calibrated capture?

A pseudo-label is inferred by a model after the footage already exists, with no independent check against ground truth. A label from calibrated, synchronized capture — 3D hand pose, wrist trajectories, action segments — is measured or annotated directly at the time of recording and checked against a written spec in QA before delivery.

Are pseudo-labels less reliable than captured labels?

A pseudo-label is only as accurate as the model or heuristic that produced it, and it carries no independent verification. A captured or annotated label is checked against the footage and the spec directly. The two are not mutually exclusive — a directly-labeled dataset can be blended with a wider pool of pseudo-labeled video to add scale.

When should you use pseudo-labels in pretraining?

Pseudo-labeling is most useful for adding scale from otherwise-unlabeled footage at the broad base of a pretraining mixture, where volume matters more than per-clip precision. Where a model needs a more specific, verified action signal — for mid-training, post-training, or coverage of a particular failure mode — a directly captured and annotated label gives stronger ground truth.

Does Firsthand deliver pseudo-labels?

No. Firsthand captures and annotates action data — 3D hand pose, wrist trajectories, contact events, and action segments — directly at the time of recording on calibrated, synchronized capture hardware, rather than inferring an action label from unlabeled footage after the fact.