What Is the Aria Digital Twin Dataset?

ADT is introduced in Aria Digital Twin: A New Benchmark Dataset for Egocentric 3D Machine Perception (arXiv 2306.06362), a paper by nine Meta Reality Labs authors. The abstract describes an egocentric dataset captured with Aria glasses and extensive object, environment and human level ground truth. The release holds 200 sequences of real-world activities by Aria wearers in two real indoor scenes, with 398 object instances, 324 stationary and 74 dynamic. The authors argue that existing datasets for augmented reality address specific sub-problems rather than the whole space, and they aim to set a standard for evaluating egocentric machine perception: 3D object detection and tracking, scene reconstruction and understanding, sim-to-real learning and human pose prediction. This page draws only on the paper, the Project Aria dataset page, the official documentation and the Project Aria Tools repository. ADT is built for augmented-reality perception research, so robot learning is Not stated in the sources read, and none of them describe robot actions.

Per the paper, the two spaces, an apartment and an office, were digitized before any recording. Rooms were scanned with a high-resolution scanner, each object was scanned with a 3D scanner, and materials came from photogrammetry tuned against real photographs. The furnished spaces were scanned again, and a motion-capture system was integrated with the digitized space and synchronized with the Aria glasses, so objects and wearers were tracked while egocentric data was recorded. Markers sit on dynamic objects and on the wearer. Synchronization uses SMPTE timecode, with a measured average accuracy under 10 microseconds in the authors’ own tests. Activities were designed in advance. The apartment has five single-person activities (room decoration, meal preparation, work, object examination, room cleaning) and three dual-person activities (partying, room cleaning, dining table cleaning); the office has object examination. Each activity has 10 to 50 sequences. The project page says recordings were made in Meta offices in North America by fully consented researchers in controlled environments. This is a purpose-built instrumented capture, not footage from the wild. Face anonymization is Not stated in the sources read.

Per the abstract, each sequence has raw data from two monochrome camera streams, one RGB camera stream and two IMU streams, complete sensor calibration, ground truth with continuous 6DoF device poses, object 6DoF poses, 3D eye gaze vectors, 3D human poses, 2D image segmentations and depth maps, and photo-realistic synthetic renderings. The project page adds 2D bounding boxes and lists two internal-facing eye-tracking cameras. All poses share one scene reference frame for every sequence in the same space. Segmentation, depth and 2D boxes are rendered from the scene model plus the tracked poses rather than drawn by hand, and occlusion between objects is handled in that process. Gaze comes from eye-tracking images at 30 Hz, processed into a per-frame direction vector, with depth from the intersection with scene objects. Human pose uses a marker-based skeleton template, and the paper says the raw marker positions are also released. Per the documentation, downloadable 3D object models in .glb arrived in September 2024, and the project page gives their geometric accuracy as 5 mm with 4k textures.

The paper also measures its own object-tracking system: average errors of 6.83 mm in translation and 1.29 degrees in rotation, from about ten frames per dynamic object held within arm’s length. Treat that as the authors’ validation of this rig, not a property of any other data.

The abstract names the research problems ADT targets: 3D object detection and tracking, scene reconstruction and understanding, sim-to-real learning and human pose prediction, plus new perception tasks for augmented reality. To demonstrate it, the paper evaluates existing methods on 2D object detection and segmentation, 3D object detection and image-to-image translation, the last using the synthetic twin of each sequence. Its reading is that models pretrained on common image datasets perform poorly on this egocentric data, which it attributes to fast ego motion and sub-optimal viewpoint. Scores are not reproduced here and none are ranked. The documentation points to an ADT Grand Challenge and the project page to ATEK, a framework for training and evaluating models on Aria data; the sources read give no further detail on either.

Both are research datasets recorded with Aria glasses, and Firsthand is part of neither. Per the existing Aria Everyday Activities entry, AEA holds 143 daily activity recordings across five indoor locations, with trajectories, a semi-dense point cloud, per-frame 3D eye gaze and speech transcription. ADT instead concentrates on two digitized spaces, with motion-capture ground truth for objects and wearers, depth, segmentation and synthetic renderings of each sequence. Hand pose annotations are Not stated in the sources read for either dataset.

The first comparison table sets only what each source or site entry states, so a cell where a source is silent says so. Neither dataset is ranked here, and this page does not claim either suits robot training.

ADT is a public research release, not a service. Firsthand is not affiliated with its authors and makes no claim of use, training or compatibility with it. If you need your own tasks and environments, the site states custom egocentric collection to a written spec, fast and on demand: first-person human video with 3D hand pose at 21 joints per hand and a per-joint visibility flag, wrist trajectories and action segments, captured with a head-mounted camera plus two wrist cameras on one hardware-triggered clock, alongside depth and IMU. The only language field is one task instruction per episode.

Confirm a spec on a Spec Pilot, a small fast first batch. The median from signed spec to first validated batch is 48 hours where coverage is deep and three to four weeks where it is thin. Billing is per validated hour; delivery is RLDS, LeRobot, WebDataset or HDF5. If contact signals matter, ask for the separate pilot-ready tactile glove tier. This page makes no claim that Firsthand provides digital-twin reconstructions, object 6DoF ground truth, eye-gaze data, motion-capture human pose, synthetic renderings or robot action data.

What is the Aria Digital Twin dataset?

A published egocentric dataset recorded with Aria glasses in two digitized indoor scenes, with ground truth for objects, the environment and the wearer, introduced in a 2023 Meta Reality Labs paper.

How large is Aria Digital Twin?

The abstract states 200 sequences and 398 object instances (324 stationary, 74 dynamic) in two scenes. The project page says about 400 minutes; the documentation says 236 sequences.

What ground truth does it include?

Device and object 6DoF poses, 3D eye gaze vectors, 3D human poses, 2D segmentation, depth maps and photo-realistic synthetic renderings, per the abstract; the project page adds 2D bounding boxes.

What is Aria Digital Twin used for?

Per the abstract, evaluating egocentric machine perception: 3D object detection and tracking, scene reconstruction, sim-to-real learning and human pose prediction. Robot learning is Not stated in the sources read.

How is it different from Aria Everyday Activities?

Per the site entry, AEA is 143 daily activity recordings in five locations. ADT is 200 sequences in two digitized spaces with motion-capture ground truth for objects and wearers, depth and synthetic renderings.

Does Firsthand use Aria Digital Twin, and can I request a custom dataset?

Firsthand is not affiliated with the ADT authors and makes no claim of use, training or compatibility. You can request a custom egocentric dataset: write a spec, confirm it on a Spec Pilot, pay per validated hour.