What Is 3D Diffuser Actor?

3D Diffuser Actor is introduced in 3D Diffuser Actor: Policy Diffusion with 3D Scene Representations (arXiv 2402.10885) by Tsung-Wei Ke, Nikolaos Gkanatsios and Katerina Fragkiadaki of Carnegie Mellon University; the project page lists CoRL 2024. The authors say it unifies diffusion policies, which learn robot action distributions, with 3D robot policies, which build 3D scene features from camera views using sensed depth.

Firsthand is not affiliated with the 3D Diffuser Actor authors and makes no claim that any model, 3D Diffuser Actor included, was trained on Firsthand data. Firsthand captures human egocentric video, not robot data: custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license. Keyposes, robot action labels, teleoperated trajectories, success or failure labels, rollouts and extra language annotations are not Firsthand deliverables, and point clouds are not a stated deliverable.

Per the abstract, 3D Diffuser Actor is a neural policy with a novel 3D denoising transformer that fuses the 3D visual scene, a language instruction and proprioception to predict the noise in noised 3D robot pose trajectories. The paper says it predicts a whole trajectory at once, non-autoregressively, by iterative denoising.

The authors say diffusion handles multimodality, since a task can often be done in several ways, while 3D scene representations support spatial reasoning.

Per the paper, each observation is one or more posed RGB-D images. A pre-trained CLIP ResNet50 featurizes each view, and depth with camera intrinsics and extrinsics lifts each feature patch to a 3D position, giving 3D scene tokens. Proprioception is one more token, and a CLIP language encoder supplies the language tokens.

A 3D relative denoising transformer applies relative 3D self-attention among the 3D tokens and cross-attention to the language tokens. It outputs noise estimates for end-effector translation and rotation, plus the gripper open or close state.

The paper evaluates on the RLBench and CALVIN benchmarks and on 12 real-world tasks with a Franka Emika arm, using 15 demonstrations per task. The arXiv abstract reports, as the authors’ result, absolute RLBench gains of 18.1% on a multi-view setup and 13.1% on a single-view setup.

The project page and README state 16.3% for the multi-view gain, so versions differ; check the current paper. These are the authors’ figures, not Firsthand’s, and this page repeats no other scores.

The authors list three: it requires camera calibration and depth information, like all 3D policies; the RLBench and CALVIN tasks are quasi-static; and it is on average slower than non-diffusion policies. The paper reports 10 Hz control on CALVIN with an NVIDIA 2080 Ti. The GitHub repository lists the code under the MIT License.

The table below sets it beside four site entries. Relations are limited to what the paper, project page and README state, so anything else is Not stated. This page ranks no models.

Per the paper, it imitates demonstration trajectories of observations and end-effector poses with a task language instruction. Firsthand states human egocentric capture with stereo-derived depth, 21-joint hand pose and one task instruction per episode. Whether those streams suit this model is Not stated, and no Firsthand data is claimed to have trained it.

When a program needs real human egocentric demonstrations that public data does not cover. Firsthand does not build or train models. Confirm the fields on a Spec Pilot, a small fast first batch; the median from signed spec to first validated batch is 48 hours where coverage is deep, three to four weeks where thin.

What is 3D Diffuser Actor?

A published robot manipulation policy (arXiv 2402.10885) that combines diffusion policies with 3D scene representations. A 3D denoising transformer fuses 3D scene tokens, a language instruction and proprioception to predict noise in end-effector trajectories. It is third-party research.

How is it different from a 2D diffusion policy?

Per the authors, it replaces per-image 2D features with 3D scene tokens built from depth and camera parameters, and in their ablation the 2D version is largely outperformed.

What camera data does it need?

Posed RGB-D images. The authors list camera calibration and depth information as a requirement, as with all 3D policies.

Is the 3D Diffuser Actor code available?

Yes. The authors publish github.com/nickgkan/3d_diffuser_actor under the MIT License. Check the repository before relying on it.

Does Firsthand use 3D Diffuser Actor, and can I request a custom dataset?

Firsthand is not affiliated with the authors and claims no use, training or compatibility. Keyposes, robot action labels, teleoperated trajectories, rollouts and success or failure labels are not Firsthand deliverables. You can request a custom dataset.