What Is Assembly101?

Assembly101 is introduced in Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities (arXiv 2203.14712), with an official project page and an official GitHub download-scripts repository. The authors say that existing multi-step activity datasets mostly follow a strictly ordered recipe or a script, or are curated from online instructional videos, and that few focus on goal-oriented activities outside the kitchen. They name three aspects they say are under-represented: goal-oriented free-style procedures, rich sequence variation, and synchronized static and egocentric viewpoints. Participants work without fixed instructions, so the recordings contain natural variation in action ordering, mistakes and corrections. Assembly101 is a video-understanding dataset. The sources read do not describe it as robot data.

Assembly101 is third-party research. Firsthand is not affiliated with its authors or their institutions, and makes no claim that Assembly101 uses Firsthand data or hardware or that any model was trained on Firsthand data. The site states custom egocentric collection to a written spec, fast and on demand, delivering first-person human video with 3D hand pose at 21 joints per hand and a per-joint visibility flag, calibrated stereo depth and 6-DoF IMUs at 200 Hz. The only language data tied to Firsthand capture is one task instruction per episode. Firsthand captures first-person human video, not robot or teleoperation data.

Per the paper, the authors built a desk rig with eight RGB cameras at 1920 × 1080 resolution and four monochrome cameras at 640 × 480 resolution. The RGB cameras are mounted on a scaffold around the desk, 5 overhead and 3 on the side. The monochrome cameras sit on the four corners of a custom-built headset worn by the participants and provide the egocentric views. All cameras are synchronized with SMPTE timecode and geometrically calibrated with a fiducial to sub-pixel accuracies. The paper also says the egocentric head pose is tracked relative to the fixed views. Participants are recorded standing, and taller participants are asked to sit so that their hands and the toy stay visible in all camera views.

The paper reports 53 adults, 28 males and 25 females, each asked to work with six toys in an hour-long recording session, though the final number varies with their speed. The sequences feature 101 unique toys from 15 vehicle categories. Only an image of the fully assembled toy was placed on the table; the authors gave no instructions and specified no part ordering. In the final protocol participants first disassemble a completed toy and then reassemble it. The paper reports 362 disassembly-assembly sequences, each recorded from 12 viewpoints, totalling 4321 videos and 513 hours of footage, with an average sequence duration of 7.1 ± 3.4 minutes.

The paper labels actions at two granularities, each with start and end times. A fine-grained action is a hand-object interaction based on a single verb or movement and an interacting object or toy part. The paper defines 1380 fine-grained labels from 24 verbs and 90 objects, which include 5 tools together with the hand, and reports more than 1M fine-grained instances. Coarse actions merge several fine-grained actions into the attaching or detaching of a vehicle part. The paper defines 202 coarse actions from 11 verbs and 61 objects and reports more than 100K coarse segments. Fine-grained actions may overlap with each other, while coarse actions are contiguous.

Beyond action labels, the paper annotates the coarse assembly segments with a parallel set of labels, “correct”, “mistake” and “correction”, and annotates each participant’s skill level from 1 (worst) to 5 (best). It reports 18M 3D hand poses, with 21 keypoints per hand in world coordinates. The paper describes these as estimated by hand tracking on the four monochrome egocentric cameras with a modified version of MegATrack. The README says the released pose files hold 2D and 3D hand poses at 60 fps and camera extrinsics for all 12 views. Natural-language annotation by the authors is Not stated in the sources read.

The paper benchmarks three action understanding tasks, recognition, anticipation and temporal segmentation, and proposes a new task of detecting mistakes. The authors say the recording format and annotations let them investigate generalization to new toys, cross-view transfer, long-tailed distributions, and pose versus appearance. They describe the motivation as smart assistants and augmented reality hardware that help people through assembly and disassembly. The paper uses 60%, 15% and 25% of the videos for training, validation and test, and says test ground truths would be withheld for online leaderboards. The project page later states that test labels have been released and that the challenge has ended.

The paper reports a domain gap between fixed and egocentric views, which it presents as a challenge for studying paired egocentric and third-person actions. Training robot policies on Assembly101 is Not stated in the sources read, and this page draws no conclusion about how it would perform as pretraining data for any robot model. For the robotics framing of hierarchical tasks, see the long-horizon task and task decomposition entries below. For egocentric human data used with robot data, see EgoMimic.

Assembly101 and custom egocentric capture answer different questions. Assembly101 is a laboratory desk-rig dataset of one task family, toy vehicle assembly, with fixed cameras and a custom headset, designed for action understanding. Custom egocentric capture is a data service: paid, consented contributors recording first-person video to a written spec, with fields and format set in advance. This page makes no ranking between them.

The comparison table below sets only what the official sources state against what the site states, and where a source is silent the cell says so. One difference matters for planning. The README states that Assembly101 is licensed under the Creative Commons Attribution-NonCommercial 4.0 International License, and that under its NonCommercial term you may not use the material for commercial purposes. A team with a commercial program should read the licence itself. The project page also says the data has been migrated from Google Drive to Hugging Face.

When a program needs tasks, objects, environments or people that an existing dataset does not cover, specified in writing, and the data has to be usable for the program’s own purpose. For multi-step manual work, write the task, the environment and the fields you need into the spec, including the per-episode task instruction and action segments the site states. If contact signals matter, ask for the separate pilot-ready tactile glove tier, time-synced to a head-mounted stereo camera by hardware timestamps and an IMU.

Confirm the spec on a Spec Pilot, a small fast first batch. Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks. Billing is per validated hour, and delivery is in the format your training stack already reads: RLDS, LeRobot, WebDataset or HDF5.

What is Assembly101?

A published dataset of 4321 videos of people assembling and disassembling 101 take-apart toy vehicles, recorded from 8 static and 4 egocentric viewpoints, with action segments, mistake labels and 3D hand poses.

How many views does Assembly101 have?

Per the paper, 12: 8 static RGB views and 4 egocentric monochrome views, synchronized with SMPTE timecode and geometrically calibrated.

What are the 3D hand poses in Assembly101?

The paper reports 18M 3D hand poses with 21 keypoints per hand, estimated by hand tracking on the four egocentric cameras.

Is Assembly101 a robot dataset?

Not stated in the sources read. The paper and README frame it around action recognition, anticipation, temporal segmentation and mistake detection, not robot policy training.

Can Assembly101 be used commercially?

The official README states a Creative Commons Attribution-NonCommercial 4.0 International License and that the material may not be used for commercial purposes.

Does Assembly101 include language annotations?

Not stated in the sources read. The authors describe verb and object action labels, mistake labels, skill levels and 3D hand poses.

Does Firsthand use Assembly101, or is it affiliated with its authors?

No. Firsthand is not affiliated with the Assembly101 authors or their institutions, and makes no claim that Assembly101 uses Firsthand data or hardware or that any model was trained on Firsthand data.

Can I request a custom egocentric dataset?

Yes. Write the tasks, environments, fields and format into a spec, confirm it on a Spec Pilot, and billing is per validated hour. Firsthand captures first-person human video, not robot or teleoperation data.