Egocentric Video Data for Inverse Dynamics and Latent Action Models

An inverse dynamics model (IDM) works backward from an observed state transition, meaning what changed between two points in a video or trajectory, to estimate the action that most plausibly caused it. A latent action model learns a compressed representation of what changed between two frames, inferred directly from footage without ever being told what the underlying action was. NVIDIA’s GR00T N1 paper (arXiv 2503.14734) uses inverse-dynamics pseudo-labeling together with latent actions so that unlabeled egocentric human video can contribute action-relevant signal at the wide base layer of its data pyramid.

Training data for these models is really two things. The first is the video itself: first-person footage in which hands, objects and their consequences are visible from one frame to the next. The second is whatever action ground truth exists alongside the video, which a team can use to fit, sanity-check or evaluate an IDM or a latent action encoder against. This page is about a collection that supplies both together.

Firsthand’s stated position, on its glossary pages for the inverse dynamics model and for latent actions, is that it does not rely on pseudo-labeling or latent actions to supply action signal, because its action data is captured and annotated directly at the time of recording. This page reads that position from the buyer’s side: if your team trains IDMs or latent action models, which captured fields can you use, which can you not, and how would you ask for them? The site states no recipe for how any team should train an IDM, so this page describes only what a collection can supply.

The glossary states the underlying problem plainly. Most egocentric human video carries no robot action label, because no robot was involved in capturing it: there is no joint angle, gripper command or wrist trajectory recorded alongside the footage. Without IDM or latent-action techniques, that footage is limited to general visual pretraining. With them, the same footage can also inform a model’s action representation before any robot-specific data enters the pyramid.

EgoScale (arXiv 2602.16710) is the clearest example on this site of how far that approach can scale. It describes a 20,854-hour corpus of action-labeled egocentric human video whose labels came from off-the-shelf SLAM and hand-pose estimation applied after capture, and reports a log-linear scaling law and a 54% success improvement on a 22-DoF hand. Those are the paper’s figures, not Firsthand’s: Firsthand claims no relationship with NVIDIA or the EgoScale authors, collected none of that data, and the site states no measured effect of its own fields on any IDM or latent action model.

The practical reason to commission a collection rather than reuse a public corpus is control. Public or off-the-shelf data is fixed by whoever published it, while a custom collection is written to your skill spec: your tasks, your environments, your streams, and your accept criteria. For a team whose IDM or latent action model has to work on a specific kind of manipulation, in specific settings, that control decides whether the video in the training set actually contains the transitions the model needs to learn from.

The glossary also notes that these approaches are not mutually exclusive. A base layer with direct action labels can be blended with a wider pool of unlabeled video processed through an IDM or latent-action technique, which is consistent with how the GR00T N1 pyramid combines layers of differing label density. A custom collection can be the directly labeled layer in that mix.

Every field named here is one the site already states as delivered. Hand pose is delivered at 21 joints per hand in metric 3D, with a per-joint visibility flag so occluded fingers are marked rather than guessed. For an IDM, that is the most direct description of what the hands did between two frames, and the visibility flag tells you which joints to trust or mask in any given pair.

Wrist trajectories are part of the standard episode schema and describe the path of each wrist through the task. The head and both wrist IMU streams, 6-DoF at 200 Hz, carry the motion record between video frames. The IMU streams are raw sensor data: they are not a camera pose or a SLAM trajectory, and this page does not present them as one.

Action segments divide each episode into start- and end-timestamped spans, each carrying start_ns, end_ns, verb and noun. For IDM work they provide a time-indexed vocabulary of what was happening, which is useful for choosing frame pairs, balancing training examples across actions, and evaluating whether a model’s inferred action changes at the right moments. Contact events mark the moment the hand meets an object, giving a labeled point at which the physical state of the scene changes.

Depth is delivered as a metric stream (depth_mm) alongside RGB, and a task instruction is paired to each episode on the same clock as the video. Per-frame instance masks, at the depth resolution of 848×480, are delivered where the spec calls for them. The table below maps each field to a possible IDM or latent-action use and to its delivery field name. The “use” column is our framing of how a team might apply the field; the site states no result for any of these uses.

One decision stays with you: the action space your IDM predicts. The site states no canonical action space and does not say what a LeRobot action field carries for a given order, so write the target action representation into the spec before capture rather than discovering a mismatch at delivery.

A latent action model is trained without an explicit action label by design, so a collection does not feed it labels in the usual sense. What a collection can supply is controlled, in-distribution video of the tasks you care about, written to your spec instead of scraped, plus captured fields that serve as an outside reference for what the learned latents should be tracking.

Three uses follow from the delivered fields, each offered as a possibility rather than a documented result. A team can compare learned latent codes against wrist trajectories and 3D hand pose to check whether the latent space follows real hand motion. It can compare changes in the latent signal against action-segment boundaries and contact events to see whether the model notices the moments a person would. And it can hold out collected episodes as an evaluation set, since their action segments and hand pose were recorded rather than inferred by the model under test.

Because the same episodes carry depth, IMU and a task instruction, a team can also ask whether latents learned from RGB alone differ from what the other streams describe. The site states no train and evaluation split, so if you want one, ask for it in the spec.

Firsthand claims no novel latent-action scheme and no IDM of its own. It does not ship a trained inverse dynamics model, a latent action encoder, a pseudo-labeling pipeline or a learned action vocabulary. What it delivers is footage and captured fields. Turning those into an IDM or a latent action model is your team’s work.

It does not deliver robot action labels. Firsthand captures first-person human video, not robot or teleoperation data, so there are no joint angles, gripper commands or end-effector actions in the data. Any mapping from human hand motion to a robot’s action space is outside what the site states.

It does not deliver a head or camera 6-DoF pose or a SLAM trajectory. The head IMU is a raw sensor stream, and no camera-pose trajectory is stated as a deliverable. EgoScale’s labels came from SLAM and hand-pose estimation; Firsthand’s delivered fields are different, and teams that need a camera trajectory should state that need and expect it to be quoted separately or not offered, because the site does not say.

It does not deliver narration or dense captions. The one language field the site states is a task instruction paired to each episode. Speech audio is not described as a deliverable either, and the site states that speech is scrubbed of identifiers at ingest. Also not stated: any label accuracy figure, any inter-annotator agreement figure, a field path for the wrist trajectories and the task instruction on the pages checked for this one, and which streams the phone and 360 capture tiers carry beyond what the spec fixes.

Start from the transitions your model needs to see. Write the tasks as a skill spec with the verbs and nouns you want in the action segments, because those are the fields each segment carries. Name the environments (the commercial program covers retail, warehouse, commercial kitchen, automotive service, tool use and assembly), the objects the task involves, and what counts as a good attempt.

Then choose streams and labels. The site describes three capture tiers, a phone in a chest mount, a 360 head mount, and the full instrumented rig, and states that not every order needs every stream. Say whether you need depth, IMU, instance masks, contact events and the task instruction, and write the action space your model predicts so the delivery format can be mapped to it. If you want a held-out evaluation set or per-country coverage reported on the data card, write that in too, because the site states neither as a default.

Finally, write the accept criteria and the license scope before capture, including whether you want exclusivity on the footage, the spec or both. The spec is quoted for reach, timeline and price before any capture begins. Most programs begin with a Spec Pilot, a small first batch that proves the spec and accept criteria on real footage before you scale. The guide on how to write a skill spec covers the format in more detail.

Acceptance is measured against signed accept criteria written into the spec before capture. Episodes are machine-checked against the reject criteria and then a human reviewer checks each one against the task spec. An episode is accepted only if RMS reprojection stays under 0.28 px and calibration drift is within tolerance; otherwise it goes to the reject log, which ships with every batch. Rejected items are recaptured at our cost, and only footage that clears the criteria is billed as a validated hour.

Timing matters more for IDM data than for most uses, because the label has to refer to the frame transition it describes. Wrist cameras, the depth stream and the IMUs are hardware-triggered onto one PTP clock domain, and the site states a sync tolerance of 1.12 ms median, 1.94 ms p99 and a 2.0 ms reject ceiling. Annotation timestamps are aligned to every other stream, so start_ns and end_ns on an action segment refer to the same clock as the video frames.

Each batch arrives with its data card, the consent artefact IDs listed per episode, the signed accept criteria it was measured against and the reject log. Faces, licence plates and screens are detected, redacted and human-reviewed. What the site does not state is a label-level accuracy metric for hand pose, contact events or segment boundaries. If your IDM evaluation depends on one, agree how it will be measured in the spec and ask what evidence the pilot batch can provide.

Data is delivered in the format your stack already reads: RLDS, LeRobot, WebDataset, HDF5, zarr or Rerun .rrd, or raw streams with metadata, with JSONL available for annotation-only delivery when labels are being added to video you already have. Every format is a projection of one canonical episode schema with four groups, meta, streams, annotations and calibration, and readable converter source ships alongside the data so the schema can be retargeted to your own loader.

In a LeRobot delivery, the head and wrist views appear as observation.images.head and observation.images.wrist_l and wrist_r, with an observation.state vector that packs the 2×21×3 hand joints first and then the 24×3 body joints, and an action field. Depth is not embedded in the mp4 video and is pulled in separately. Check the formats pages for the layout of the format you will load, and confirm in the spec which fields feed your action target.

The license is perpetual, irrevocable, worldwide and buyer-owned, non-exclusive by default. Permitted use includes training, evaluating, fine-tuning and deploying models, including commercial models. Weights derived from the data are yours, with no royalty, revenue share or reporting obligation. Redistribution of the footage itself is not permitted, while models derived from it can be distributed without limit. Exclusivity is available per skill line or per environment, priced as a multiplier, and the order form governs any specific agreement.

From signed spec to first validated batch, the timeline is a median of 48 hours where coverage is deep and three to four weeks where it is thin and contributors need to be recruited first. After the first batch, on-demand top-ups run against the same accept criteria, so you can add more attempts, environments or tasks without rewriting the spec.

Pricing is quoted per engagement against your spec and billed per validated hour. This page states no hour count, no throughput and no customer, because the site does not, and it makes no promise about how long a negotiated exclusivity term takes. Settling the scope early, in the Spec Pilot conversation, keeps contract terms from becoming the slow part.