What Is EgoSchema?

EgoSchema is introduced in EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding (arXiv 2308.09126) by Karttikeya Mangalam, Raiymbek Akshulakov and Jitendra Malik of UC Berkeley. The abstract calls it a very long-form video question-answering dataset and benchmark for evaluating how well modern vision and language systems understand long video. It is derived from Ego4D, with over 5000 human curated multiple choice question-answer pairs spanning over 250 hours of real video, and each question asks for the correct answer among five options based on a three-minute clip. The arXiv page shows one version from 17 August 2023, and the PDF is marked as a preprint under review, so a publication venue is Not stated in the sources read. Robot learning is also Not stated in the sources read.

Per the paper, EgoSchema does not record new footage. Its RGB clips are sourced from Ego4D, which the paper describes as over 3670 hours of RGB video with over 3.85 million narration instances, where narrators describe everything the camera wearer, written as C, does. Stage I of the pipeline keeps non-overlapping three-minute clips that each have at least 30 human-annotated narrations. The paper lists a limitation: the clips inherit Ego4D’s egocentric video biases. The project page says a file connecting questions to the original Ego4D metadata is in development. Participant counts, locations and devices for the EgoSchema subset are Not stated in the sources read.

Per the paper, a four-stage pipeline lightens the load on human annotators. After Stage I filtering, Stage II gives each clip’s narrations to a large language model, which generates three questions per clip, then a correct answer and four wrong answers for each, making five options. The authors report that only GPT-4, Bard and Claude gave good quality at a high enough rate, after about 85 prompts. Stage III discards badly formatted outputs and applies a blind filter that removes questions a language model can answer without the video narrations.

Stage IV is manual. In a first round, annotators check that each question is well formed, the answer is correct, the distractors are wrong, and the temporal certificate length is at least 30 seconds. That round cuts admissible questions by about 4x to 5x, and more than 97% of the questions that pass also pass a second round of re-curation. The README says answers are submitted as an integer from 0 to 4 per question.

The authors argue that clip length alone does not capture temporal difficulty, so they introduce temporal certificate sets. A certificate is the minimum set of sub-clips needed to convince a human verifier that the marked answer is correct, and certificate length is the total duration of those sub-clips. The paper calls tasks with certificates of about 100 seconds very long-form and reports a median EgoSchema certificate of about 100 seconds, estimated on a random sample of 100 clips.

The abstract states intrinsic temporal lengths over 5.7x longer than the second closest dataset and 10x to 100x longer than any other video understanding dataset. Other passages in the paper give slightly different ratios, so quote the one you cite.

The abstract says models with several billion parameters score under 33% on the multiple-choice task, where random is 20%, while humans reach about 76%. The paper’s zero-shot table lists FrozenBiLM (1.2B) at 26.4% and 26.9%, VIOLET (198M) at 19.9% and 19.6%, mPLUG-Owl (7.2B) from 20.0% to 31.1%, and InternVideo (478M) from 31.4% to 32.1%, depending on the number of frames. The README adds that closed-source models with 100B+ parameters score under 40%.

Human accuracy ranges from 67.0% under a one-minute limit to 76.2% when watching the video before reading the text. These are 2023 sources, not a current leaderboard; any later score is Not stated in the sources read.

The README describes three ways to download the videos: the Kaggle egoschema-public competition, a Wasabi download script, or a zipped Google Drive file. It releases correct answers for only 500 questions, because EgoSchema is meant as a zero-shot benchmark. Scoring on the full set is by submitting predictions to the Kaggle leaderboard or a validation server, which reports accuracy over all 5031 questions.

The abstract says the data and zero-shot evaluation code are open-sourced under the Ego4D license, and the paper says that licence allows public use of the video and text data for research and commercial purposes. The repository’s own licence terms are Not stated in the sources read. Not legal advice; read the licence files.

EgoSchema is an evaluation set, not a source of new footage: its clips and narrations come from Ego4D, which the site’s Ego4D entry describes as a large public dataset and benchmark suite. EgoLife is its own week-long recording with an EgoLifeQA benchmark, and Charades-Ego is a dataset of paired third- and first-person videos of scripted indoor activities. All are third-party research, and Firsthand is not part of any of them. The table below sets only what each official source or site entry states.

EgoSchema is a public benchmark, and Firsthand is not affiliated with its authors. Firsthand makes no claim that its data trained, was evaluated on or is compatible with EgoSchema or any model scored on it. EgoSchema rests on narrations and multiple-choice questions over three-minute clips; the Firsthand site does not state question-answer annotations, video captions or narrations, evaluation or benchmark datasets, eye gaze or third-person video.

What the site states is custom egocentric collection of real human demonstrations to a written spec, fast and on demand: head and wrist camera video, depth, 3D hand pose at 21 joints per hand with a per-joint visibility flag, wrist trajectories, action segments and contact events. The only language field is one task instruction per episode. Confirm a spec on a Spec Pilot, a small fast first batch. The median from signed spec to first validated batch is 48 hours where coverage is deep and three to four weeks where it is thin. Billing is per validated hour; delivery is RLDS, LeRobot, WebDataset or HDF5.

What is EgoSchema?

A very long-form video question-answering benchmark derived from Ego4D: over 5000 human curated multiple-choice questions spanning over 250 hours of video, each with five options and a three-minute clip, per arXiv 2308.09126.

How hard is EgoSchema for models and humans?

Per the abstract, models with several billion parameters scored under 33% where random is 20%, and humans about 76%. These are 2023 results; any later score is Not stated in the sources read.

How are the questions made?

A language model generates three questions per three-minute clip from Ego4D narrations, each with one correct and four wrong answers. Rule-based and blind filtering and two rounds of manual curation follow.

Can EgoSchema be used commercially?

The abstract says the data and evaluation code are open-sourced under the Ego4D license, and the paper says that licence allows research and commercial use of the video and text data. The repository’s own licence is Not stated in the sources read. Read the licence files; this is not legal advice.

Is the full answer key public?

No. The README releases correct answers for only 500 questions and says the entire answer file will not be made public; scoring on the full set is by submitting predictions to the Kaggle leaderboard or a validation server.

Does Firsthand use EgoSchema, and can I request a custom dataset?

Firsthand is not affiliated with EgoSchema’s authors and makes no claim of use, training, evaluation or compatibility. You can request a custom egocentric dataset: write a spec, confirm it on a Spec Pilot, pay per validated hour.