What Is Act3D?

Per the abstract, many manipulation tasks require high spatial precision in end-effector pose prediction, which typically demands high-resolution 3D feature grids that are expensive to process, so most manipulation policies operate directly in 2D. Act3D is the authors’ answer: it lifts 2D pre-trained features to 3D using sensed depth, attends to them to compute features for sampled 3D points, and samples 3D point grids in a coarse to fine manner. A 3D action map here is the map of candidate end-effector locations the policy scores in the workspace.

Firsthand is not affiliated with the Act3D authors and makes no claim that any model, Act3D included, was trained on Firsthand data. Firsthand captures human egocentric video, not robot data: custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license. 3D feature fields, point clouds, end-effector keyposes, robot action labels, teleoperated trajectories, success or failure labels, rollouts and language annotations beyond one task instruction per episode are not Firsthand deliverables.

Per the authors’ project page, Act3D is a Transformer manipulation policy trained with supervised learning to predict a 6-DoF end-effector pose from one or more RGB-D images, a language instruction and proprioception. Its key idea is to detect end-effector poses in the workspace by learning 3D perceptual representations of free space with arbitrary spatial resolution, via recurrent coarse-to-fine 3D point grid sampling and featurization.

In the abstract’s words, the model samples 3D point grids in a coarse to fine manner, featurizes them using relative-position attention, and selects where to focus the next round of point sampling. That is how it computes 3D action maps efficiently instead of building one dense high-resolution grid.

Per the project page, Act3D featurizes multi-view RGB images with a pre-trained 2D backbone and lifts them to 3D using depth, giving a multi-scale 3D scene feature cloud. It then iteratively predicts 3D foci of attention in free space, samples 3D point grids in their vicinity, and featurizes those points using relative cross-attention to the scene feature cloud, language tokens and proprioception.

The project page says a detection Transformer head picks the 3D point for the next best end-effector position, and the model regresses the rotation, end-effector opening and a collision avoidance action. The abstract credits relative spatial attention, large-scale vision-language pre-trained 2D backbones and weight tying across coarse-to-fine attentions as the components the authors tested in ablations.

Multi-task means one policy handles many manipulation tasks, conditioned on a language instruction. Per the abstract, Act3D achieves a 10% absolute improvement over the previous state-of-the-art 2D multi-view policy on 74 RLBench tasks, and a 22% absolute improvement with 3x less compute over the previous state-of-the-art 3D policy. The project page also describes a multi-task, multi-variation setting with 18 tasks and 249 variations proposed by PerAct.

These are the authors’ figures, not Firsthand’s, and this page repeats no other scores. The authors state they test learning from demonstrations in simulation and the real world.

The abstract points to a project website with code and videos, and the authors’ GitHub repository (zhouxian/act3d-chained-diffuser) describes a language-conditioned policy learner proposed in two papers: Act3D and ChainedDiffuser. This page is about Act3D only.

The README says the authors train on a remote cluster and evaluate locally with RLBench and PyRep, and that dataset generation lives in a data_preprocessing folder. A license for the repository was not stated in the sources read for this page, so check the repository before relying on it.

The table below sets Act3D beside four site entries. Relations are limited to what the Act3D abstract, project page and README state, so anything else is Not stated. This page ranks no models.

Per the project page, Act3D learns from demonstrations with supervised learning, from RGB-D images, a language instruction and proprioception, with robot end-effector pose targets. Firsthand states human egocentric capture with stereo-derived depth, 21-joint hand pose and one task instruction per episode. Whether those streams suit Act3D is Not stated, and no Firsthand data is claimed to have trained it.

When a program needs real human egocentric demonstrations that public data does not cover. Firsthand does not build or train models. Confirm the fields on a Spec Pilot, a small fast first batch; the median from signed spec to first validated batch is 48 hours where coverage is deep, three to four weeks where thin.

What is Act3D?

A published manipulation policy transformer (arXiv 2306.17817) that represents the robot workspace as a 3D feature field with adaptive resolution and computes 3D action maps of high spatial resolution. It is third-party research.

What is a 3D feature field in Act3D?

Per the authors, Act3D lifts 2D pre-trained features to 3D using sensed depth, then attends to them to compute features for sampled 3D points, sampled in a coarse to fine manner so resolution adapts to the task.

What does multi-task robotic manipulation mean here?

One policy handles many manipulation tasks, conditioned on a language instruction. The authors evaluate on RLBench, including 74 tasks per the abstract, and report their own results; this page quotes no other scores.

Is the Act3D code available?

The authors point to a project website and publish github.com/zhouxian/act3d-chained-diffuser, which also covers ChainedDiffuser. A license was not stated in the sources read, so check the repository before relying on it.

Does Firsthand use Act3D, and can I request a custom dataset?

Firsthand is not affiliated with the Act3D authors and claims no use, training or compatibility. 3D feature fields, point clouds, keyposes, robot action labels, teleoperated trajectories, rollouts and success or failure labels are not Firsthand deliverables. You can request a custom dataset.