What Is RT-Trajectory?

RT-Trajectory is introduced in RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketches (arXiv 2311.01977) by Jiayuan Gu, Sean Kirmani, Ted Xiao and colleagues, submitted on 3 November 2023. The abstract says “we propose a policy conditioning method using such rough trajectory sketches, which we call RT-Trajectory”, which names it. The project page lists Google DeepMind, UC San Diego, Stanford University and Intrinsic as affiliations and ICLR 2024 (Spotlight) as the venue. Hindsight means the training sketches are drawn after the fact from demonstrations that were already recorded.

Firsthand is not affiliated with the RT-Trajectory authors and makes no claim that any model, RT-Trajectory included, was trained on Firsthand data. Firsthand captures human egocentric video, not robot data: custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license. Trajectory sketches, camera calibration files, multi-view rigs, 3D labels, reward labels, success or failure labels, rollouts, latent action labels, planning annotations and language annotations beyond one task instruction per episode are not Firsthand deliverables.

Per the abstract, language-conditioned policies struggle to generalize to new tasks: a policy trained on pick-and-place will not fold, even when the arm trajectory is similar. The authors’ insight is to describe the task as a rough trajectory sketch instead, one detailed enough to express low-level motion but coarse enough that the policy can read it against what the camera sees.

The project page says the sketches are the task specification for an RT-1 policy backbone. They are trained on hindsight sketches and tested on sketches produced at inference time.

Per the project page, for each demonstration episode the authors take the recorded robot proprioception, get the 3D end-effector centre position in the robot base frame, and project it into camera space with the known camera extrinsic and intrinsic parameters. The pixel positions are joined by straight lines on a blank image, with green circles where the gripper closes and blue ones where it opens.

Colour grading adds more. The red channel carries the normalized time step, and the green channel carries normalized height. The authors name two forms, RT-Trajectory (2D) and RT-Trajectory (2.5D), where 2.5D adds the height information to reduce depth ambiguity.

The project page lists four sources: human drawings made in a simple GUI over the robot’s initial camera image, human demonstration videos, code written by an LLM, and an image-generation model. For videos, the authors estimate hand poses from first-person single-hand videos and convert them to robot tool poses.

These are the authors’ own setups. Whether egocentric video from any other source works with RT-Trajectory is Not stated in the sources read.

The comparison table below sets RT-Trajectory beside four site entries. In short, RT-1 is the policy backbone, RT-H predicts language motions, trajectory data is a recorded stream, and TraceVLA overlays point-tracked traces on the input image.

This page ranks no models. It repeats no success rates, benchmark scores or parameter counts, because the numbers were not read as single stated values in the sources.

The project page says the authors train on the RT-1 demonstration dataset: 551 instructions, 8 manipulation skills on 17 household kitchen items, and 73,334 real robot demonstrations collected by manual teleoperation. The authors evaluate on 7 new skills. These are the authors’ figures on their own setup.

Firsthand states human egocentric capture with stereo-derived depth, 21-joint hand pose and one task instruction per episode. Whether those streams could produce sketches is Not stated, and no Firsthand data is claimed to have trained RT-Trajectory.

When a program needs real human egocentric demonstrations that public data does not cover. Firsthand does not build or train models. Confirm the fields on a Spec Pilot, a small fast first batch; the median from signed spec to first validated batch is 48 hours where coverage is deep, three to four weeks where thin.

What is RT-Trajectory?

A published method (arXiv 2311.01977) that conditions a robot policy on a rough trajectory sketch of the intended motion, so the policy can perform tasks that language-conditioned and goal-conditioned policies find hard. It is third-party research.

What are hindsight trajectory sketches?

Per the project page, sketches drawn from recorded demonstrations: the end-effector path is projected into the camera view, with markers where the gripper closes and opens. They are made after the demonstration, hence hindsight.

How does RT-Trajectory get a sketch for a new task?

The project page lists human drawings, human demonstration videos, code written by an LLM and an image-generation model as ways to produce sketches at inference time.

Are the RT-Trajectory weights or code available?

Not stated in the abstract or the project page, which are the sources read. This page links no repository; check the paper before assuming any release.

Does Firsthand use RT-Trajectory, and can I request a custom dataset?

Firsthand is not affiliated with the RT-Trajectory authors and claims no use, training or compatibility. Trajectory sketches, camera calibration files, rollouts, reward labels and success or failure labels are not Firsthand deliverables. You can request a custom dataset.