What Is the IndustReal Dataset?

IndustReal is introduced in IndustReal: A Dataset for Procedure Step Recognition Handling Execution Errors in Egocentric Videos in an Industrial-Like Setting (arXiv 2310.17323) by Tim J. Schoonbeek, Tim Houben, Hans Onvlee, Peter H.N. de With and Fons van der Sommen of Eindhoven University of Technology and ASML Research, published at WACV 2024. The abstract argues that action recognition for procedural tasks gives no measure of success for actions. It defines the task of procedure step recognition (PSR), meaning recognizing the correct completion and order of procedural steps, and presents IndustReal alongside it. The abstract says IndustReal contains procedural errors such as omissions as well as execution errors, and the authors describe it as the first dataset to also include execution errors, for example the wrong type of nut. This page draws on the paper, the official project page and the official GitHub repository.

Twenty-seven participants, each with a signed consent form and approval from the institution’s Ethical Review Board, performed assembly and maintenance procedures on a 3D-printed construction-toy car based on the STEMFIE toy, which has 36 part models. Two procedures are defined: building the car from scratch, and a maintenance task that replaces the long braces with short braces on the rear chassis. Printed instructions are subgoal-oriented, so participants may build subassemblies and choose the order between subgoals. Each person received correct instructions plus one or two instructions containing errors, and mistakes made with correct instructions were annotated as well. Recording used a HoloLens 2 headset with the HL2SS library, on a white desk against a white background. The dataset has 84 videos; the paper’s comparison table gives 5.8 hours and the project page says 6 hours.

PSR asks a system to estimate, at each time t, which procedure steps a person has correctly completed so far, given sensor inputs and a description of the procedural actions. The paper says this definition allows real-time operation because it does not need a full recording. A step counts as completed when its component is correctly installed, which includes tightening a nut. The paper contrasts PSR with action recognition, which recognizes which actions are performed rather than completed.

IndustReal has 38 errors, 14 of them only in the validation and test sets, so the data can test robustness to unseen mistakes. It has 724 correct step completions and 38 incorrect ones, and 35 videos (42%) contain a missing or incorrectly completed step. Errors range from minor, such as installing the wheels without washers, to large, such as forgetting a wheel. The paper notes that explicit detection of incorrect step execution is not part of the PSR task definition; incorrect states are annotated for qualitative analysis and future work.

Labels cover three tasks. Action recognition has 75 fine-grained action classes and 9,273 instances, built from the 12 verbs used in MECCANO plus component names. The average action lasts 1.9 ± 1.4 seconds and 24.2% of instances overlap another action. Assembly state detection divides the car into 11 components and labels 22 defined states plus 27 error states with bounding boxes in COCO format, on 26.9K frames (13% of the total), 3,569 of which show error states. PSR labels give the frame where a step completes and the new assembly state, in a correct-only set and a set that includes incorrect steps. The split is 12 training, 5 validation and 10 test participants, made by participant rather than by video.

These are the paper’s own baseline results. For action recognition, the best result in its Table 2 is an MViTv2 ensemble over RGB, visible-light and stereo images, with 66.45% top-1 and 88.43% top-5. For assembly state detection, YOLOv8-m reaches mAP 0.838 on boxed frames when trained on real plus synthetic images, falling to 0.641 across entire videos. For PSR the best baseline, B3, scores a procedure order similarity of 0.797 and an F1 of 0.883 over all recordings, and 0.731 and 0.816 on recordings with errors. The authors conclude the baseline fails to generalize to execution errors.

The GitHub repository TimSchoonbeek/IndustReal says the dataset holds 84 videos of 27 participants, with part geometries and trained weights for the best models in the paper. Each recording has RGB, stereo left and right, depth and ambient-light folders, gaze, hand and head-pose CSV files, and action recognition, object detection and PSR label files. The README describes hand tracking as 52 coordinates per hand per row. The paper and README disagree on two image sizes: the paper gives RGB 1280×720 and stereo 480×620, while the README gives 1080×720 and 640×480, so check the data itself.

The paper publishes 3D models of all parts to encourage training on synthetic data, and its assembly state baseline used 100K synthetic samples generated with Unity Perception. The README states that IndustReal is released under the Apache 2.0 license; any further terms on the data download are Not stated in the sources read. Check them before commercial use; this is not legal advice.

IndustReal answers research questions: one construction-toy car, 27 participants, a consistent white desk and background, a HoloLens 2, and labels for steps, states and errors. Custom capture is a data service to a written spec, and this page makes no ranking between them. Field names overlap but the kinds differ. IndustReal’s hand and gaze data come from HoloLens 2 tracking; the site states 3D hand pose at 21 joints per hand with a per-joint visibility flag and lists a gaze_target annotation field where the spec calls for it, but names no eye-tracking device, sampling rate or accuracy, so no default gaze stream is claimed.

Firsthand makes no error-labelling, mistake-detection or procedure-step-recognition benchmark claim, and no claim that its data trained, was used with or is compatible with IndustReal, its baselines or any other benchmark, model or robot. IndustReal’s synthetic-data baselines are its authors’ choice; Firsthand data is real capture, never synthetic, generated or simulated. The site states no robot action or joint data, teleoperation data, simulation data, visual question answering data or video captions, and the only language field it states is one task instruction per episode.

Firsthand collects real egocentric demonstrations on demand, to a written spec. For assembly work, write the task, parts, tools and environment into the spec. The rig is a head-mounted global-shutter camera plus two wrist cameras, with active-stereo depth and 3x IMU on one hardware-triggered PTP clock. Episodes carry 3D hand pose at 21 joints per hand with a per-joint visibility flag, wrist trajectories, contact events, frame-accurate action segments and one task instruction per episode. Error labels are not a stated deliverable.

Confirm the spec on a Spec Pilot and pay per validated hour. The site states a 48-hour median turnaround where coverage is deep and three to four weeks where it is thin. Delivery is RLDS, LeRobot, WebDataset or HDF5, with a buyer-owned commercial license. Request a custom dataset below.

What is the IndustReal dataset?

A published egocentric, multimodal dataset in which 27 participants performed assembly and maintenance procedures on a construction-toy car while wearing a HoloLens 2, labeled for action recognition, assembly state detection and procedure step recognition.

What is procedure step recognition?

Per the paper, a task that estimates which procedure steps a person has correctly completed so far, recognizing the correct completion and order of steps rather than only which actions are being performed.

What does IndustReal record?

RGB, stereo visible-light and long-throw depth images, plus gaze, hand and head-pose tracking, from a HoloLens 2 at 10 fps, with depth at 5 fps. The paper and README differ on two image sizes, so check the data itself.

Does IndustReal include mistakes?

Yes. The paper reports 38 errors, 14 of them only in the validation and test sets, covering procedural errors such as omissions and execution errors such as the wrong type of nut.

Can IndustReal be used commercially?

The README states IndustReal is released under the Apache 2.0 license; any further terms on the data download are Not stated in the sources read. Check the project page and GitHub repository directly before commercial use; this is not legal advice.

Does Firsthand use IndustReal, and can I request a custom dataset?

Firsthand is not affiliated with IndustReal’s authors and makes no claim of use, training or compatibility. Firsthand data is real human egocentric capture. You can request a custom egocentric dataset: write a spec, confirm it on a Spec Pilot, pay per validated hour.