What Is RVT (Robotic View Transformer)?

RVT is introduced in RVT: Robotic View Transformer for 3D Object Manipulation (arXiv 2306.14896) by Ankit Goyal, Jie Xu, Yijie Guo, Valts Blukis, Yu-Wei Chao and Dieter Fox of NVIDIA; the official GitHub README lists CoRL 2023 (Oral). The authors say methods that build an explicit 3D representation perform better than those relying only on camera images, but explicit representations such as voxels cost a lot of compute. Multi-view here means the policy reasons over several rendered images of the same scene at once.

Firsthand is not affiliated with the RVT authors and makes no claim that any model, RVT included, was trained on Firsthand data. Firsthand captures human egocentric video, not robot data: custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license. Rendered virtual views, point clouds, end-effector keyposes, robot action labels, teleoperated trajectories, success or failure labels, rollouts and language annotations beyond one task instruction per episode are not Firsthand deliverables.

Per the abstract, two key features of RVT are an attention mechanism that aggregates information across views and re-rendering of the camera input from virtual views around the robot workspace. The authors say this decouples the images fed to the transformer from the real camera images.

Per the paper, that lets them render from useful viewpoints, such as directly above the table, without real-world camera placement limits. Because the multi-view input comes from re-rendering, RVT can also run with a single sensor camera, as in the authors’ real-world experiments.

Per the paper, the input is a language description of the task, the current visual state from RGB-D camera(s) and the gripper state (open or closed). The authors first reconstruct a point cloud, then render it from virtual viewpoints anchored at the robot base. Each view has 7 channels: RGB, depth and the (x, y, z) world coordinates of each point, which link pixels across views.

A joint transformer with eight self-attention layers processes the views and language; in the first four, image tokens attend only within the same image. It outputs an 8-dimensional action: a 6-DoF target end-effector pose, a 1-DoF gripper state and a binary collision indicator for the low-level motion planner, predicted at the next key-frame. Translation comes from per-view heatmaps back-projected into 3D.

Per the abstract, a single RVT model works across 18 RLBench tasks with 249 task variations, achieving 26% higher relative success than PerAct, training 36X faster for the same performance and reaching 2.3X the inference speed. The abstract also says RVT can perform a variety of real-world manipulation tasks with just a few (about 10) demonstrations per task.

In the real-world setup, per the paper, a person specified gripper target poses by moving the robot arm by hand while an RGB-D stream was recorded. These are the authors’ figures, not Firsthand’s, and this page repeats no other scores.

The authors say RVT, like voxel-based methods such as PerAct, requires calibration of the camera extrinsics to the robot base. They also say view specification could be optimized or learned from data, and that RVT struggled on marker-related real-world tasks, likely due to sparse and noisily sensed point clouds.

The GitHub README lists the code under the Nvidia Source Code License and the pretrained models under CC-BY-NC-SA-4.0. The same repository also covers RVT-2 (RSS 2024); this page is about the original RVT.

The table below sets RVT beside three site entries. Relations are limited to what the RVT paper, project page and README state, so anything else is Not stated. This page ranks no models.

Per the paper, RVT trains with behavior cloning on expert demonstrations, each pairing RGB-D observations and gripper state with the robot actions and a language description of the task. Firsthand states human egocentric capture with stereo-derived depth, 21-joint hand pose and one task instruction per episode. Whether those streams suit RVT is Not stated, and no Firsthand data is claimed to have trained it.

When a program needs real human egocentric demonstrations that public data does not cover. Firsthand does not build or train models. Confirm the fields on a Spec Pilot, a small fast first batch; the median from signed spec to first validated batch is 48 hours where coverage is deep, three to four weeks where thin.

What is RVT (Robotic View Transformer)?

A published multi-view transformer for 3D object manipulation (arXiv 2306.14896). It re-renders camera input from virtual views around the robot workspace, attends across the views and predicts a target end-effector pose. It is third-party research.

What is virtual view re-rendering in RVT?

Per the authors, RVT reconstructs a point cloud from RGB-D input and renders it from virtual viewpoints around the robot base, so the transformer’s input is decoupled from the real camera images.

How many demonstrations does RVT need?

The abstract says about 10 demonstrations per task in the real world. Those are the authors’ figures, not Firsthand’s.

Is the RVT code available?

Yes. The authors publish github.com/NVlabs/RVT under the Nvidia Source Code License, with pretrained models under CC-BY-NC-SA-4.0. Check the repository before relying on it.

Does Firsthand use RVT, and can I request a custom dataset?

Firsthand is not affiliated with the RVT authors and claims no use, training or compatibility. Rendered virtual views, point clouds, keyposes, robot action labels, teleoperated trajectories, rollouts and success or failure labels are not Firsthand deliverables. You can request a custom dataset.