What Is the EgoLife Dataset?

EgoLife is introduced in EgoLife: Towards Egocentric Life Assistant (arXiv 2503.03803) by Jingkang Yang and colleagues at S-Lab, Nanyang Technological University, LMMs-Lab and other institutions; the official GitHub README lists CVPR 2025. The abstract describes a project to develop an egocentric life assistant that accompanies and enhances personal efficiency through AI-powered wearable glasses. To lay the foundation, six participants lived together for one week and continuously recorded their daily activities, including discussions, shopping, cooking, socializing and entertainment. The abstract calls the result the EgoLife Dataset, a 300-hour egocentric, interpersonal, multiview and multimodal daily-life dataset with intensive annotation. This page draws only on the paper, the official project page, the official GitHub repository and the Hugging Face dataset card. It is human recording data: robot learning is Not stated in the sources read.

Per the paper, the data was collected over seven days with six volunteers living in a custom-designed environment called the EgoHouse. Each participant wore Meta Aria glasses and captured multimodal egocentric video, and the paper’s overview figure gives approximately eight hours of egocentric video and signals per participant per day. Fifteen GoPro cameras recorded third-person views from multiple angles, and millimeter-wave radars (two mmWave devices) provided spatial and motion data. Participants were asked to organize an Earth Day party on the second-to-last day, so the footage includes meetings, rehearsals, cooking, decorating, shopping and sightseeing, with recording permission obtained in places such as shopping malls. The authors say each pair of glasses recorded at least six hours per day while participants were awake. Because of storage limits, recordings were structured into three-hour segments, and the glasses were collected every three hours for upload, a process of about one hour. The project page lists video, gaze and IMU data from the glasses, plus 3D scans of the house and participants.

The abstract states 300 hours. The paper’s Table 1 instead lists 266 hours, six clips and 44.3 hours per clip, and a note under Table 2 says that value is derived from 266 hours of retained footage across six participants. The project page says about 50 hours of footage per participant (7 days x 8 hours). These figures are not reconciled in the sources read, so quote the one you cite.

Table 1 also lists 400K+ captions. The primary language is Chinese: all annotations (transcripts, captions and questions) are primarily in Chinese and translated into English, and the paper notes a later one-day recording session with predominantly English speaking.

The paper’s data pipeline synchronizes video, audio and IMU from the glasses and GoPro cameras, then applies privacy protection (EgoBlur), dense captioning and transcription. For transcripts, the authors ran speech recognition and open-source speaker diarization, then had reviewers refine one track per participant, keeping only the speech audible to that person. For captions, annotators narrated 5-minute clips, producing 361K short narrations that averaged 2.65 seconds. A language model merged these into 25K captions, and GPT-4o then combined them with sampled frames and transcripts into verified visual-audio captions.

EgoLifeQA is the benchmark built on top. The paper describes five question types: EntityLog, EventRecall, HabitInsight, RelationMap and TaskMaster. Generated candidates were filtered by annotators down to 500 multiple-choice questions per participant, 3,000 in total, each needing evidence from at least five minutes earlier. Table 2 reports 997 questions with a certificate length below 2 hours and 2,003 above it. Evaluation scores are not reproduced here.

The authors present EgoButler, made of EgoGPT and EgoRAG. Per the paper, EgoGPT is a 7B omni-modal model that captions each 30-second clip from visual and audio input. It is fine-tuned on LLaVA-OneVision using EgoIT-99K, a set of question-answer data derived from nine existing egocentric datasets (43.16 hours, 99.48K QA), and personalized on EgoLife Day 1. EgoRAG is a retrieval-augmented module that builds a memory bank of captions summarized at hourly and daily levels and retrieves evidence for long-context questions.

The authors also list limits. EgoGPT’s speech understanding is incomplete, and fine-tuning on Day 1 can lead it to confuse people in later days. EgoRAG makes a single retrieval pass without multi-step reasoning. The paper says the task remains challenging with large room for improvement.

All four are third-party research datasets, and Firsthand is not part of any of them. EgoLife centres on one week of continuous multi-person daily life in a single house, while the site’s Ego4D, Aria Everyday Activities and Nymeria entries describe broad worldwide footage, short scripted indoor sequences and whole-body motion respectively. The table below sets only what each source or site entry states; see the linked entries for detail.

The GitHub README says the EgoLife video is released on Hugging Face, along with the EgoIT-99K dataset and the first versions of the EgoGPT and EgoRAG code. The Hugging Face dataset card organizes the videos by day, and links a model fine-tuned on the dataset. The repository lists its licence as Other, and its README says the project is licensed under the S-Lab license. The licence terms of the dataset itself, and whether commercial use is permitted, are Not stated in the sources read. Not legal advice; read the licence files.

EgoLife is a public research release, and Firsthand is not affiliated with its authors. Firsthand makes no claim that its data trained, was used with or is compatible with EgoLife, EgoGPT, EgoRAG or EgoLifeQA. EgoLife is built around week-long life recording with third-person cameras and language annotations; the Firsthand site does not state multi-day continuous life logging, eye gaze, third-person video, audio transcripts or video captions.

What the site states is custom egocentric collection of real human demonstrations to a written spec, fast and on demand: head and wrist camera video, depth, 3D hand pose at 21 joints per hand with a per-joint visibility flag, wrist trajectories, action segments and contact events. The only language field is one task instruction per episode. Confirm a spec on a Spec Pilot, a small fast first batch. The median from signed spec to first validated batch is 48 hours where coverage is deep and three to four weeks where it is thin. Billing is per validated hour; delivery is RLDS, LeRobot, WebDataset or HDF5.

What is the EgoLife dataset?

A multimodal egocentric daily-life dataset from six participants who lived together for one week wearing Meta Aria glasses, with synchronized third-person video. The abstract of arXiv 2503.03803 states 300 hours; Table 1 lists 266.

What is EgoLifeQA?

A benchmark of long-context, life-oriented multiple-choice questions built on EgoLife: 500 per participant, 3,000 in total, across five types (EntityLog, EventRecall, HabitInsight, RelationMap, TaskMaster).

What are EgoGPT, EgoRAG and EgoButler?

EgoGPT is the authors’ 7B omni-modal model for clip-level captioning and question answering, EgoRAG is a retrieval module over hourly and daily caption summaries, and EgoButler is the system combining the two.

What language is EgoLife in?

Primarily Chinese. The paper says all annotations (transcripts, captions and questions) are primarily in Chinese and translated into English, and a later one-day session was recorded with predominantly English speaking.

Can EgoLife be used commercially?

Not stated in the sources read. The GitHub repository lists its licence as Other and its README says the project is licensed under the S-Lab license; the dataset’s own terms are not stated. Read the licence files; this is not legal advice.

Does Firsthand use EgoLife, and can I request a custom dataset?

Firsthand is not affiliated with EgoLife’s authors and makes no claim of use, training or compatibility. You can request a custom egocentric dataset: write a spec, confirm it on a Spec Pilot, pay per validated hour.