What Is the Charades-Ego Dataset?

Charades-Ego is described in Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videos (arXiv 1804.09626), a 2018 paper by Gunnar Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi and Karteek Alahari, with affiliations at Carnegie Mellon University, Inria and the Allen Institute for Artificial Intelligence. It describes the egocentric side of a dataset first introduced in Actor and Observer, which links the first- and third-person video understanding domains. The abstract gives 68,536 activity instances in 68.8 hours of first and third-person video, and says the dataset shares activity classes, scripts and methodology with the original Charades dataset, which adds 82.3 hours of third-person video with 66,500 activity instances. The authors’ motivation is that if first and third-person understanding can be linked, billions of easily available third-person videos could improve egocentric video understanding. The project page lists 7,860 videos of daily indoor activities collected through Amazon Mechanical Turk, with 68,536 temporal annotations for 157 action classes. This page draws only on the paper, the two project pages and the official GitHub repository for the earlier Actor and Observer code. Robot learning is Not stated in the sources read, and none of them describe robot actions.

Per the paper, the authors followed the Hollywood in Homes approach used for Charades: crowd workers recruited over the internet recorded themselves performing a given script of activities. Each Amazon Mechanical Turk worker recorded two videos, one acting out the script from a third-person view and another acting out the same script in the same way with a camera fixed to their forehead. The authors call this a compromise between a synchronized lab setting that records both views at once and scalability, so the two videos are separate recordings of one script rather than simultaneous views. Workers could hold the camera to their forehead and use one hand, or build their own head mount and use two hands; the paper quotes the earlier Actor and Observer work as saying over 60% of submitted videos showed both hands with a home-made mount. Most scripts come from Charades, with 1000 more collected the same way, and 78.7% of scripts are shared with the Charades training set. The paper cites 112 actors, with videos from rooms in 112 homes across the world, performing the same 157 activities as Charades. The camera model, resolution and frame rate are Not stated in the sources read, and so are face anonymization, eye gaze and hand pose. This is scripted crowd-sourced recording, not footage from the wild.

Per the abstract, Charades-Ego has temporal annotations and textual descriptions. The paper reports 68,536 activity instances, or 8.72 activities per video on average, across the 157 Charades activity classes, in 34.4 hours of third-person video and 34.4 hours of corresponding first-person video. The project page gives 7,860 videos, which matches the paper’s 80/20 split into 6,167 training videos and 1,693 test videos, with no subject in both sets. The smallest number of examples per category is 52 in training and 24 in testing.

For annotation, the paper copies activities from original Charades videos that share a script and runs the third-person videos through video-level annotation. Workers then annotated both views of the same subject at the same time, with the third-person video used to obtain better annotations for the first-person video. Each task was one video pair and 5 activities on average. The authors report that annotator consensus for egocentric videos is 4.2% lower than for third-person videos (77.0% versus 72.8%). The Charades page lists 41,104 labels for 46 object classes and 27,847 textual descriptions for the original Charades; whether Charades-Ego carries object labels is Not stated in the sources read.

The abstract says the temporal annotations and textual descriptions make it suitable for egocentric video classification, localization, captioning and new tasks that use the cross-modal nature of the data. The paper itself presents classification baselines only, on version 1.0, scored as video-level mean average precision with ResNet-152 models pretrained on the original Charades third-person data. In the authors’ own comparison, a model trained on first-person labels scores 27.1 on the first-person test set, against 19.5 for third-person training, 28.2 for first- and third-person training and 7.1 for random. Baselines for localization and captioning are Not stated in the sources read.

These are the authors’ baselines for this dataset, reproduced as stated. They are not ranked against anything else and say nothing about Firsthand data. The GitHub README for the earlier Actor and Observer code notes that Table 3 of that CVPR 2018 paper used version 0 of the dataset, so check the dataset version before comparing any score.

Both pair first-person video with another view, and Firsthand is part of neither. Per the existing Ego-Exo4D entry, its official sources state simultaneously captured egocentric and exocentric video of skilled activities such as sports, music, dance and bike repair, from 740 participants in 13 cities, 1,286 hours combined, with calibrated, time-synchronized cameras. Charades-Ego instead pairs two separate recordings of the same script by crowd workers at home, one third-person and one with a forehead-mounted camera, covering everyday indoor activities.

The first comparison table sets only what each source or site entry states, so a cell where a source is silent says so. Neither dataset is ranked here, and this page does not claim either suits robot training.

Charades-Ego is a public research release, not a service. Firsthand is not affiliated with its authors and makes no claim of use, training or compatibility with it. Firsthand captures egocentric video only: the site states it does not capture third-person or fixed external-camera footage, so it offers no paired first- and third-person video. If you need your own tasks and environments, the site states custom egocentric collection to a written spec, fast and on demand: first-person human video with 3D hand pose at 21 joints per hand and a per-joint visibility flag, wrist trajectories and action segments, captured with a head-mounted camera plus two wrist cameras on one hardware-triggered clock, alongside depth and IMU. The only language field is one task instruction per episode.

Confirm a spec on a Spec Pilot, a small fast first batch. The median from signed spec to first validated batch is 48 hours where coverage is deep and three to four weeks where it is thin. Billing is per validated hour; delivery is RLDS, LeRobot, WebDataset or HDF5. If contact signals matter, ask for the separate pilot-ready tactile glove tier. This page makes no claim that Firsthand provides paired or exocentric video, scripted crowdsourced home recordings, eye-gaze data, robot action data or video captions.

What is the Charades-Ego dataset?

A published dataset of paired third-person and first-person videos of scripted indoor activities, recorded by crowd workers, described in a 2018 paper by Sigurdsson, Gupta, Schmid, Farhadi and Alahari.

How large is Charades-Ego?

The paper states 68,536 activity instances in 68.8 hours, 34.4 hours each of third- and first-person video. The project page gives 7,860 videos and 157 action classes.

How was Charades-Ego collected?

Amazon Mechanical Turk workers recorded themselves acting out a script twice, once third-person and once with a forehead-fixed camera. The two videos are separate recordings, not simultaneous views.

What is Charades-Ego used for?

Per the abstract, egocentric video classification, localization, captioning and tasks that use the cross-modal first- and third-person pairing. Robot learning is Not stated in the sources read.

How is it different from Ego-Exo4D?

Per the site entry, Ego-Exo4D is simultaneously captured, time-synchronized egocentric and exocentric video of skilled activity. Charades-Ego pairs two separate scripted recordings of everyday indoor activities.

Does Firsthand use Charades-Ego, and can I request a custom dataset?

Firsthand is not affiliated with the Charades-Ego authors and makes no claim of use, training or compatibility, and it captures egocentric video only. You can request a custom egocentric dataset: write a spec, confirm it on a Spec Pilot, pay per validated hour.