What Is the MECCANO Dataset?

MECCANO is introduced in MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain (arXiv 2209.08691) by Francesco Ragusa, Antonino Furnari and Giovanni Maria Farinella of FPV@IPLab at the University of Catania and Next Vision s.r.l., a spin-off of the university. The project page lists the paper as published in Computer Vision and Image Understanding in 2023 and asks users to also cite an earlier WACV 2021 paper on the first MECCANO video dataset. The abstract says human behavior analysis is well studied in third-person vision but understudied in egocentric settings, particularly industrial ones. The introduction says the industrial domain had not been explored from a first-person view, mainly because data acquisition there is difficult given privacy issues and the need to protect industrial secrets. This page draws on the paper, the official project page and the official GitHub repository.

Twenty subjects in two countries, Italy and the United Kingdom, built a toy model of a motorbike with 49 components, a screwdriver, a wrench and an instruction booklet laid out on a light or dark table. Participants came from 8 nationalities and were aged 18 to 55. They sat, were free to use any object and the booklet, and received no other instruction. A custom headset combined an Intel RealSense SR300 for RGB and depth with a Pupil Core device for gaze, recorded together in Pupil Capture software. RGB is 1920×1080 and depth 640×480, both at 12 fps, with gaze at 200 Hz. The paper reports a constant 0.4 s offset between depth and RGB that it corrected, and keeps gaze positions with confidence of at least 0.6. Each video is one complete assembly, averaging 21.14 minutes in the paper text.

The paper annotates 8857 temporal segments with 12 verbs: take, put, check, browse, plug, pull, align, screw, unscrew, tighten, loosen and fit. Each segment has a start, a contact and an end time, and 1401 segments (15.82%) overlap another. Verbs and 20 object classes combine into 61 action classes. Active objects get bounding boxes every 0.2 seconds, 64349 frames in total. Next-active objects are boxed in frames up to 3 seconds before contact, 48024 frames with 74127 boxes, covering 75.66% of interactions. Hands get 169625 boxes over 89628 frames, pre-labeled by a hand-object detector and corrected by annotators. These are 2D boxes with a left or right label; 3D hand pose is Not stated in the sources read.

The paper benchmarks five tasks: action recognition, active object detection and recognition, egocentric human-object interaction (EHOI) detection, action anticipation and next-active object detection. The project page lists six, adding egocentric gaze estimation, and points to procedure learning annotations as a further use.

The paper splits the data into 11 training, 2 validation and 7 test videos, balanced by country and table colour; the project page describes 11 training videos and 9 validation or test videos. Baselines include SlowFast on RGB, depth and gaze variants for recognition, a hand-object detector and Faster R-CNN for active objects, and RULSTM for anticipation.

Reported Top-1 action recognition is 45.16% for RGB alone, 49.49% with depth and 49.66% with gaze added, and the paper calls the gaze gain minor. Active object detection rises from 11.17 AP to 38.14 AP after re-training the detector. Combining all five anticipation branches did not give the best result. The authors conclude the tasks are challenging for state-of-the-art methods.

The project page offers downloads of RGB videos and frames, depth frames, gaze data, action and EHOI verb temporal annotations, and hands, active-object and next-active-object bounding boxes. The GitHub repository fpv-iplab/MECCANO is described as the official repository with code and models, with instructions for PySlowFast, Detectron2 and RULSTM and pretrained baselines. Licence terms are Not stated in the sources read. Check them before any commercial use; this is not legal advice.

The paper frames MECCANO as a testbed. It argues that recognizing human-object interactions from a first-person view could help monitor machine use, guide operators and flag missed actions, and that anticipating a worker’s next actions could improve factory safety. These are motivations, not results. The paper also calls its toy motorbike task a simplification of real industrial settings. In related work it says Assembly101 was inspired by the first MECCANO version and differs in that MECCANO focuses on multimodal data. Using MECCANO to train robot policies is Not stated in the sources read.

MECCANO answers research questions: one seated assembly task, one toy, 20 participants, a research headset with RGB, depth and gaze, and verb and bounding-box labels. Custom capture is a data service to a written spec, and this page makes no ranking between them. Field names overlap but the kinds differ. MECCANO’s hand labels are 2D boxes; the site states 3D hand pose at 21 joints per hand with a per-joint visibility flag. MECCANO’s gaze comes from a Pupil Core; the site lists a gaze_target annotation field where the spec calls for it, but names no eye-tracking device, sampling rate or accuracy, so no default gaze stream is claimed.

Firsthand makes no claim that its data trained, was used with or is compatible with MECCANO, its baselines or any other benchmark, model or robot. Firsthand data is real capture, never synthetic, generated or simulated. The site states no robot action or joint data, teleoperation data, simulation data, visual question answering data or video captions, and the only language field it states is one task instruction per episode.

Firsthand collects real egocentric demonstrations on demand, to a written spec. For assembly work, write the task, parts, tools and environment into the spec. The rig is a head-mounted global-shutter camera plus two wrist cameras, with active-stereo depth and 3x IMU on one hardware-triggered PTP clock. Episodes carry 3D hand pose at 21 joints per hand with a per-joint visibility flag, wrist trajectories, contact events, frame-accurate action segments and one task instruction per episode.

Confirm the spec on a Spec Pilot and pay per validated hour. The site states a 48-hour median turnaround where coverage is deep and three to four weeks where it is thin. Delivery is RLDS, LeRobot, WebDataset or HDF5, with a buyer-owned commercial license. Request a custom dataset below.

What is the MECCANO dataset?

A published multimodal egocentric dataset in which 20 participants built a toy motorbike while wearing a custom headset that recorded RGB video, depth maps and gaze, labeled for human behavior understanding tasks.

What modalities does MECCANO include?

Per the paper, RGB video at 1920×1080, depth maps at 640×480 (both 12 fps) and gaze at 200 Hz, recorded with an Intel RealSense SR300 and a Pupil Core device.

What tasks does MECCANO support?

The paper benchmarks action recognition, active object detection and recognition, egocentric human-object interaction detection, action anticipation and next-active object detection. The project page also lists egocentric gaze estimation.

Is MECCANO a real industrial dataset?

No. The paper calls it industrial-like and describes the toy motorbike task as a simplification of real industrial settings. It cites privacy and industrial secrets as reasons industrial data is hard to acquire.

Can MECCANO be used commercially?

Licence terms are Not stated in the sources read. Check the project page and GitHub repository directly before commercial use; this is not legal advice.

Does Firsthand use MECCANO, and can I request a custom dataset?

Firsthand is not affiliated with MECCANO’s authors and makes no claim of use, training or compatibility. Firsthand data is real human egocentric capture. You can request a custom egocentric dataset: write a spec, confirm it on a Spec Pilot, pay per validated hour.