Glossary
What Is PerAct (Perceiver-Actor)?
Last reviewed: October 2026
Short answer
PerAct (Perceiver-Actor) is a multi-task Transformer for robotic manipulation (arXiv 2209.05451) from Shridhar, Manuelli and Fox. It is a language-conditioned behavior-cloning agent that encodes a language goal and RGB-D voxel observations with a Perceiver Transformer and outputs discretized actions by detecting the next best voxel action. It is third-party research; Firsthand is not affiliated.
01
What is PerAct, and what does voxel-based language-conditioned mean?
PerAct is introduced in Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation (arXiv 2209.05451) by Mohit Shridhar, Lucas Manuelli and Dieter Fox. The project page lists the University of Washington and NVIDIA as affiliations and CoRL 2022 as the venue. Voxel-based means the observation is a 3D grid rebuilt from RGB-D sensors rather than a 2D image, and language-conditioned means a natural-language goal is part of the input.
02
Where does Firsthand fit next to PerAct?
Firsthand is not affiliated with the PerAct authors and makes no claim that any model, PerAct included, was trained on Firsthand data. Firsthand captures human egocentric video, not robot data: custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license. Voxel grids, keyframe annotations, robot action labels, teleoperated trajectories, success or failure labels, rollouts and language annotations beyond one task instruction per episode are not Firsthand deliverables.
03
How does PerAct work?
Per the abstract, PerAct is a language-conditioned behavior-cloning agent for multi-task 6-DoF manipulation. It encodes language goals and RGB-D voxel observations with a Perceiver Transformer and outputs discretized actions by “detecting the next best voxel action”.
The authors contrast this with 2D-image frameworks: the voxelized 3D observation and action space, they say, provides a strong structural prior for learning 6-DoF actions. The project page calls the approach action-centric, with no object detectors, instance segmentors or pose estimators building the scene first.
04
What does the PerAct input and output look like?
Per the project page, the input is a language goal plus a voxel grid reconstructed from RGB-D sensors. The grid is 100×100×100, about one million voxels. Extracting 5×5×5 patches gives 20×20×20, or 8,000 embeddings. Language and voxel features form one sequence for a PerceiverIO Transformer.
Perceiver encodes this with a small set of latent vectors trained end to end. Output features predict a discretized translation, rotation, gripper open state and collision avoidance, which a motion planner can execute.
05
What tasks and data do the authors describe?
Per the abstract, the authors train a single multi-task Transformer for 18 RLBench tasks (with 249 variations) and 7 real-world tasks (with 18 variations) from just a few demonstrations per task. The project page states, as the authors’ result, that PerAct outperforms image-to-action agents by 34x and 3D ConvNet baselines by 2.8x. This page repeats no other scores.
The GitHub README lists checkpoints trained on 18 tasks with 100 demonstrations per task, and its FAQ advises 10 to 20 demonstrations for useful behavior and 50 to 100 for robustness. These are the authors’ figures, not Firsthand’s.
06
What limits do the authors state?
The README says some tasks, such as push_buttons, are not solvable by PerAct because it has no memory, and points to Appendix L of the paper for other limitations. The code is under Apache License 2.0.
07
How does PerAct compare with 3D-VLA, VoxPoser, BridgeVLA and a language-conditioned policy?
The table below sets PerAct beside four site entries. The sources read are the PerAct paper, project page and README, so relations to later work are Not stated. This page ranks no models.
08
What data does PerAct need, and what does Firsthand state?
PerAct trains by behavior cloning on demonstrations paired with language goals, in simulation and on a real arm. Firsthand states human egocentric capture with stereo-derived depth, 21-joint hand pose and one task instruction per episode. Whether those streams suit PerAct is Not stated, and no Firsthand data is claimed to have trained PerAct.
09
When do you need fast, on-demand custom collection instead?
When a program needs real human egocentric demonstrations that public data does not cover. Firsthand does not build or train models. Confirm the fields on a Spec Pilot, a small fast first batch; the median from signed spec to first validated batch is 48 hours where coverage is deep, three to four weeks where thin.
10
PerAct at a glance, as stated in the paper, project page and GitHub README
| Item | What the sources state |
|---|---|
| Paper | Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation (arXiv 2209.05451) |
| Core idea | A language-conditioned behavior-cloning agent that detects the next best voxel action |
| Input | A language goal and a voxel grid reconstructed from RGB-D sensors |
| Output | Discretized translation, rotation, gripper open and collision avoidance, executed with a motion planner |
| Tasks in the abstract | 18 RLBench tasks (249 variations) and 7 real-world tasks (18 variations) |
| Code | github.com/peract/peract, Apache License 2.0 |
| Firsthand relationship | None |
12
PerAct beside the 3D-VLA, VoxPoser, BridgeVLA and language-conditioned policy entries
| Site entry | What the entry states | How PerAct relates (sources read) |
|---|---|---|
| Language-conditioned policy | A policy whose actions depend on a natural-language instruction as well as observations such as camera images | PerAct is a language-conditioned behavior-cloning agent, per the abstract |
| VoxPoser | A method where a language model composes 3D value maps in the observation space for zero-shot trajectories | Not stated in the sources read; PerAct is trained on demonstrations and detects voxel actions instead of composing value maps |
| 3D-VLA | A 3D generative world model that predicts goal images and point clouds to guide robot control | Not stated in the sources read; PerAct predicts no goal images or point clouds |
| BridgeVLA | A 3D VLA that projects 3D inputs into 2D images and predicts actions as 2D heatmaps | Not stated in the sources read; PerAct keeps a voxel observation and action space |
13
Frequently asked questions
- What is PerAct (Perceiver-Actor)?
- A published multi-task Transformer for robotic manipulation (arXiv 2209.05451). It encodes a language goal and RGB-D voxel observations with a Perceiver Transformer and outputs discretized actions by detecting the next best voxel action. It is third-party research.
- Why does PerAct use voxels instead of images?
- Per the authors, the voxelized 3D space provides a strong structural prior for learning 6-DoF actions, unlike 2D-image frameworks.
- How much data does PerAct need?
- The abstract says a few demonstrations per task. The README’s checkpoints use 100 per task, and its FAQ advises 10 to 20 for useful behavior and 50 to 100 for robustness. Those are the authors’ figures.
- Is the PerAct code available?
- Yes. The authors publish github.com/peract/peract under Apache License 2.0. Check the repository before relying on it.
- Does Firsthand use PerAct, and can I request a custom dataset?
- Firsthand is not affiliated with the PerAct authors and claims no use, training or compatibility. Voxel grids, robot action labels, teleoperated trajectories, rollouts and success or failure labels are not Firsthand deliverables. You can request a custom dataset.
Related terms