Glossary
What Are Neural Trajectories in Robot Learning?
Last reviewed: October 2026
Short answer
Neural trajectories are synthetic robot videos produced by a video-generation model that has been fine-tuned on real teleoperation data, then labeled with pseudo-actions for training. NVIDIA’s GR00T N1 paper (arXiv 2503.14734) describes generating 827 hours of neural trajectories from 88 hours of in-house teleoperation data — about a 10x expansion — to fill the middle of its data pyramid.
01
Why it matters
Real robot teleoperation data scales linearly with human labor, since an operator drives the robot through every trajectory. GR00T N1’s paper uses neural trajectories to get around that, generating counterfactual robot videos from novel language prompts. The tradeoff: generated video has no native action label, so the paper assigns pseudo-actions with latent-action or inverse-dynamics models.
02
How Firsthand handles it
Firsthand does not generate, label, or sell neural trajectories or synthetic robot video. It captures real egocentric human video — synchronized RGB, depth, 3D hand pose, and action-segment labels, on one PTP clock — fast, on demand, to a written spec.
03
What are neural trajectories?
A neural trajectory is a synthetic robot video produced by a video-generation model, not recorded from a real robot session. GR00T N1’s paper fine-tunes an image-to-video model on real teleoperation data, then prompts it with novel language instructions to generate "counterfactual" videos it never recorded.
The stated reason: "Robot data scales linearly with human labor, since it typically requires a human operator to teleoperate the robot to produce each trajectory."
04
Where do neural trajectories come from in GR00T N1?
GR00T N1 fine-tunes an image-to-video model on its in-house teleoperation dataset, then generates videos from existing initial frames and novel language prompts in the form "pick up {object} from {location A} to {location B}", limited to physically feasible combinations.
A commercial-grade multimodal LLM proposes those combinations, then judges 8 sampled frames per generated video; videos that do not follow the instruction are filtered out and re-captioned.
05
Where do they sit in the GR00T N1 data pyramid?
Neural trajectories sit in the middle, synthetic layer, alongside simulation trajectories — above the base of web data and egocentric human video, below the narrow top layer of real robot trajectories.
The paper fine-tunes its video model on all 88 hours of in-house teleoperation data and generates 827 hours of video, "augmenting it by around 10×."
06
How are neural trajectories labeled with actions?
Generated video carries no action signal. The paper states: "since the generated videos do not have action labels, we use either latent or inverse dynamics models (IDM) labeled actions ... and train the policy model to treat these pseudo-actions as action labels for a different embodiment."
Its IDM conditions on a current and a future frame from a trajectory and predicts the action chunk between them, then pseudo-labels the neural trajectories.
07
How do neural trajectories differ from real egocentric human video?
Both are video without native robot action labels, so GR00T N1 applies the same latent-action and inverse-dynamics labeling to each. Their origin is opposite: egocentric human video is real footage of a person doing a real task; a neural trajectory is video a model generated from a text prompt.
A generated trajectory is bounded by the real teleoperation data that trained its video model, which the paper keeps limited "to match the realistic scenario that we only have access to limited number of teleoperation data."
08
When does a team still need real, custom-collected egocentric data?
Neural trajectories target a shortage of teleoperated robot hours. They do not replace real egocentric human video, which GR00T N1 places in the earlier base layer and draws from existing human video datasets rather than generating.
A program short on egocentric coverage of a specific scene, task, or object set needs more real human-performed footage, not more generated robot video. Firsthand’s custom collection targets that gap: fast, on-demand capture to a written spec.
09
Neural trajectories vs. real egocentric human video vs. real robot data, per the GR00T N1 paper
| Neural trajectories | Egocentric human video | Real robot data | |
|---|---|---|---|
| Origin | Generated by a video model fine-tuned on real teleoperation data | Recorded from a real person performing a real task | Recorded from a real robot executing a real teleoperated trajectory |
| Pyramid layer | Middle (synthetic) | Base | Top |
| Native action label | None — pseudo-labeled with latent actions or an inverse dynamics model | None — same pseudo-labeling approach applies | Direct, from the teleoperation signal |
| Scale GR00T N1 reports | 827 hours generated from 88 hours of teleoperation (≈ 10×) | Per-dataset hours in the paper’s Table 7; not summarized as a single total | 88 hours of in-house teleoperation |
11
What the GR00T N1 paper states about neural trajectories vs. what it does not state
| Question | What the paper states | Not stated, or not restated here |
|---|---|---|
| Which video model was fine-tuned? | Open-source image-to-video models; the appendix names WAN2.1-I2V-14B fine-tuned with LoRA | Hyperparameters, sample counts and GPU cost: not restated here |
| How much video was generated? | 827 hours from 88 hours of real teleoperation (≈ 10×) | Per-task breakdown of those hours: not stated in the sections used here |
| How are actions assigned? | Latent actions or an inverse dynamics model (IDM) produce pseudo-actions | The latent-vs-IDM comparison results: not restated here |
| How is quality controlled? | A commercial-grade multimodal LLM judges sampled frames; failing videos are filtered out and re-captioned | Which LLM: not stated |
| Is every use case evaluated? | Multi-round long-horizon generation and neural trajectories of liquids and articulated objects are demonstrated | Quantitative downstream evaluation: left for future work |
12
Frequently asked questions
- What are neural trajectories in robot learning?
- Synthetic robot videos generated by a video model fine-tuned on real teleoperation data, then labeled with pseudo-actions, as described in the GR00T N1 paper (arXiv 2503.14734).
- How much data does GR00T N1 generate as neural trajectories?
- The paper fine-tunes its video model on all 88 hours of in-house teleoperation data and generates 827 hours of video, roughly a 10× expansion.
- How are neural trajectories labeled with actions if they were never teleoperated?
- Generated videos carry no action labels, so GR00T N1 uses latent actions or an inverse dynamics model (IDM) to produce pseudo-actions for a different embodiment.
- Where do neural trajectories sit in the GR00T N1 data pyramid?
- In the middle, synthetic layer alongside simulation trajectories, between the base layer of web data and egocentric human video and the top layer of real robot trajectories.
- Are neural trajectories the same as egocentric human video?
- No. Egocentric human video is real footage of a person, in the base layer. Neural trajectories are generated robot-action video in the synthetic layer. Both lack native action labels.
- Does Firsthand generate or sell neural trajectories?
- No. Firsthand captures real egocentric human video, fast and on demand, to a written spec. It does not generate, label, or sell synthetic robot video or neural trajectories.
Related terms
Sources