Glossary
What Is Trajectory Data in Robot Learning?
Last reviewed: October 2026
Short answer
Trajectory data is the recorded path of a body part over time. In robot learning it is usually the motion of a wrist or hand across a clip, stored as a time-stamped stream beside the video. Firsthand delivers wrist motion as 6-DoF IMU streams at 200 Hz and 3D hand joints per frame, collected fast, on demand, to your spec.
01
What is trajectory data in robot learning?
A trajectory is an ordered series of states over time. In robot learning it is the path a hand, wrist or end effector follows through a task, stored as its own stream next to the video that shows what the scene looked like. A policy needs it because video alone shows what happened, not the motion that produced it. Two kinds matter for egocentric data. A wrist trajectory is the coarse path of the hand: where it went and how it moved on the way to an object. A hand trajectory is the per-joint 3D position of the fingers over time. Together they stand in for the action half of a training example.
02
How is a wrist or hand trajectory captured from egocentric video?
Firsthand records both on its instrumented rig during capture, not afterwards from RGB. Wrist motion comes from a 6-DoF BMI088 IMU on each wrist, sampled at 200 Hz and delivered as streams/imu_wrist_l and streams/imu_wrist_r. The hand trajectory is delivered as 3D hand joints aligned to depth and RGB: 21 joints per hand, in metres, in the head-camera frame, with a visibility flag per joint. Every stream shares one PTP clock domain and carries int64 nanosecond timestamps, so the wrist and hand paths line up with the video, depth and action segments. Collection is fast, on demand and to a written skill spec.
03
What fields does a trajectory stream contain?
The first table lists the trajectory-related fields the site states, with their delivery names, shapes and rates.
The wrist IMU is sampled at 200 Hz, while hand_joints is indexed by the head-camera frame count T. The two series therefore run at different rates and are aligned through their nanosecond timestamps, not by sample count.
04
How is trajectory data used to train robot and humanoid policies?
NVIDIA’s EgoScale (arXiv 2602.16710) pretrains a policy on 20,854 hours of egocentric human video. It supervises the policy with relative wrist motion between consecutive timesteps, plus hand-joint actions retargeted from 21 hand keypoints. Its camera motion and hand trajectories were recovered after capture with off-the-shelf SLAM and hand-pose estimation, which the paper describes as noisy.
The same paper compares action representations and reports that a wrist-only representation performed poorly on tasks needing precise finger articulation. That is why a spec should state both wrist and hand requirements, not assume one implies the other.
Firsthand supplies the measured version of those inputs, recorded live during capture. It does not state any robot training results from its own data: Not stated.
05
How does trajectory data differ from neural trajectories and action representations?
Neural trajectories are a different thing despite the name. They are synthetic robot videos from a video-generation model fine-tuned on teleoperation data; GR00T N1 (arXiv 2503.14734) generated 827 hours from 88 hours of teleoperation data. Firsthand does not generate or sell them.
An action representation is the format the action half of a training example takes. Trajectory data is one measured input a buyer can build that representation from.
06
How do you request a custom trajectory dataset?
Write down the tasks and environments your model needs, the fields you want (wrist motion, hand joints, contact events, action segments) and the delivery format: RLDS, LeRobot, WebDataset or HDF5 / zarr. A custom collection starts from a written skill spec, typically proven on a smaller Spec Pilot, with a first validated batch in a median of 48 hours where coverage is deep. Anything marked Not stated below is set per program in the spec.
07
Trajectory-related fields Firsthand states, with delivery names and formats
| Field | What it encodes | Delivery field | Stated shape and rate |
|---|---|---|---|
| Wrist motion | 6-DoF IMU (acceleration and rotation) on each wrist | streams/imu_wrist_l, streams/imu_wrist_r | float32 [N, 6], 200 Hz, BMI088 |
| Hand joints | Per-joint 3D hand position, both hands | annotations/hand_joints | float32 [T, 2, 21, 3], metres, head-camera frame |
| Joint visibility | Whether each joint is visible | annotations/hand_visible | Per joint; shape and rate Not stated |
| Contact events | Grasp and release moments | annotations/contact_events | Shape and rate Not stated |
| Action segments | Sub-action boundaries with verb and noun | annotations/action_segments | {start_ns, end_ns, verb, noun}; 97 verbs, 512 nouns |
| Timestamps | Per-stream time on one clock | *_timestamp_ns | int64 nanoseconds, PTP domain |
09
Trajectory data compared with related terms
| Term | What it is | Where it comes from | Firsthand relationship |
|---|---|---|---|
| Trajectory data | Recorded path of a wrist or hand over time | Measured at capture | Delivers wrist IMU streams and 3D hand joints |
| Action representation | Format the action half of a training example takes | Built by the buyer or paper from inputs | Supplies measured inputs; does not define its own |
| Neural trajectories | Synthetic robot video from a fine-tuned video model | Generated (GR00T N1) | Does not generate or sell them |
| Hand pose estimation | Per-frame 3D hand and finger joints | Measured or estimated | Delivers 21 joints per hand with visibility flags |
10
What the site states and what is Not stated
| Detail | Stated on the site | Status |
|---|---|---|
| Wrist pose derived from the IMU | No | Not stated; set per program |
| Coordinate frame for wrist motion | No | Not stated; set per program |
| Accuracy, smoothing or filtering | No | Not stated; set per program |
| Head or camera 6-DoF pose, SLAM trajectories | No | Not stated |
| Wrist IMU key in RLDS, LeRobot, WebDataset | No | Not stated; set per program |
11
Frequently asked questions
- What is trajectory data in robot learning?
- The recorded path of a wrist or hand over time, stored as a time-stamped stream beside the video. It gives a policy the motion that video alone does not show.
- What is a wrist trajectory?
- The coarse path of the hand: where it went and how it moved. Firsthand records wrist motion live as a 6-DoF IMU stream on each wrist at 200 Hz. How a wrist pose is derived from it is Not stated and is set per program.
- What is hand trajectory data for robot training?
- The per-joint 3D position of the fingers over time. Firsthand delivers 21 joints per hand in metres, in the head-camera frame, with a visibility flag per joint.
- Does Firsthand deliver head or camera pose trajectories?
- The site does not state that, so none is claimed. Head or camera 6-DoF pose and SLAM trajectories are Not stated.
- Are trajectory data and neural trajectories the same?
- No. Trajectory data is measured motion. Neural trajectories are synthetic robot videos generated by a fine-tuned video model, which Firsthand does not sell.
- Which formats carry trajectory fields?
- Datasets ship in RLDS, LeRobot, WebDataset or HDF5 / zarr. The wrist IMU streams are carried natively as their own keys in HDF5 / zarr; where they sit in the other formats is Not stated and set per program.
- How can I order a custom trajectory dataset quickly?
- Request a custom dataset with your tasks, fields and format. Collection is on demand against a written spec, with a first validated batch in a median of 48 hours where coverage is deep.
Related terms