What Is the HoloAssist Dataset?

HoloAssist is introduced in HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real World (arXiv 2309.17024), presented at ICCV 2023 per the project page. It has an official project page and two GitHub repositories. The authors call it a first step toward AI assistants that perceive, reason and collaborate with humans in the real world. The sources disagree on size: the paper abstract says 166 hours and the project page says 169. Both state 350 unique instructor-performer pairs. The paper text adds 222 participants and 20 object-centric manipulation tasks on 16 objects. It is human recording data, so robot actions are Not stated in the sources read.

Per the paper, the task performer wears an AR headset (HoloLens 2, per a footnote) and completes the task while the data streams over the network to a remote instructor, who watches the egocentric video on a laptop and guides the performer verbally. Performers get only a rough task description beforehand, while instructors are briefed in detail. A client on the headset showed a hologram frame to keep task actions in view of the sensors, and a PC server wrote the data to disk, allowing longer uninterrupted sessions. The authors open-sourced the capture tool, built on the Platform for Situated Intelligence.

Per the paper, 20 tasks cover 16 objects, from a GoPro, Nintendo Switch and DSLR to a Nespresso machine, IKEA furniture and rare equipment such as a NavVis laser scanner, an ATV motorcycle and a circuit breaker. Examples include changing batteries, changing belts, furniture assembly and machine setup. The IKEA pieces have both assembly and disassembly tasks.

Third-person manual annotations add a text summary per video, coarse-grained actions (task steps, usually around 30 seconds), fine-grained actions (indivisible movements of about 1 to 2 seconds, each marked correct or a mistake and, if a mistake, whether the performer or instructor corrected it), and conversation segments with transcription and a purpose label such as correcting a wrong action. The paper builds benchmarks on mistake detection, intervention type prediction and hand forecasting, and also benchmarks action classification and anticipation. Scores are not reproduced here.

The paper argues that language-model assistants lack first-hand experience of the physical world, and that existing systems often rely on predefined instructions or treat assistance as question answering or video understanding without real interaction. From the recordings it observes that instructors often intervene proactively, before a mistake happens, that guidance is concise and grounded in the performer’s surroundings through spatial deictics, and that instructors judge whether a mistake disrupts the task.

Robot learning is Not stated as a goal in the sources read; the target is assistants that guide humans. Any robot use is the reader’s own judgement. What the data does hold is egocentric video with hand pose, depth and action segments, the kind of human manipulation signal embodied-AI teams look for.

The distinguishing feature is the two-person instruct-and-perform setting. The paper’s comparison table marks HoloAssist as collaborative and interactive, and as instructional and procedural, at 166 real video hours; Ego4D is listed at 3,670 hours with those aspects only partially included. The paper calls HoloAssist complementary to Ego4D. The Ego-Exo4D and Aria Everyday Activities columns below use only what those site entries state, each with its own sources. Firsthand is not part of any of the four.

HoloAssist is a public research release, not a service. Firsthand is not affiliated with its authors and makes no claim of use, training or compatibility with it. If you need your own tasks and environments, the site states custom egocentric collection to a written spec, fast and on demand: 3D hand pose at 21 joints per hand with a per-joint visibility flag, wrist trajectories and action segments. The only language field is one task instruction per episode.

Confirm a spec on a Spec Pilot, a small fast first batch. The median from signed spec to first validated batch is 48 hours where coverage is deep and three to four weeks where it is thin. Billing is per validated hour; delivery is RLDS, LeRobot, WebDataset or HDF5. This page makes no claim that Firsthand offers headset capture, head pose, eye gaze or two-person instructor and performer sessions.

What is the HoloAssist dataset?

A large-scale egocentric human interaction dataset in which a performer wearing a mixed-reality headset completes manipulation tasks while a remote instructor guides them verbally.

How large is HoloAssist?

The paper abstract states 166 hours from 350 unique instructor-performer pairs; the project page says 169 hours. The paper text adds 222 participants and 20 tasks.

What data streams does HoloAssist include?

Seven raw modalities: RGB, depth, head pose, 3D hand pose, eye gaze, audio and IMU, plus human-annotated text as an eighth, per the paper.

What is HoloAssist used for?

The paper builds benchmarks on mistake detection, intervention type prediction and hand forecasting, plus action classification and anticipation. Robot training is Not stated in the sources read.

Can HoloAssist be used commercially?

The project page and README say it is released under CDLAv2, described as a permissive licence. Read the licence text; this is not legal advice. Other terms are Not stated in the sources read.

Does Firsthand use HoloAssist, and can I request a custom dataset?

Firsthand is not affiliated with the HoloAssist authors and makes no claim of use, training or compatibility. You can request a custom egocentric dataset: write a spec, confirm it on a Spec Pilot, pay per validated hour.