What Is the TACO Dataset?

TACO is introduced in TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understanding (arXiv 2401.08399) by Yun Liu, Haolin Yang, Xu Si, Ling Liu, Zipeng Li, Yuxiang Zhang, Yebin Liu and Li Yi. The abstract says existing approaches for analyzing and synthesizing hand-object manipulation are mostly limited to a single hand and object because of a lack of data support. TACO is the authors’ answer: a bimanual dataset covering a large variety of tool-action-object compositions. The project page describes it as covering these combinations in real-world scenarios, and as supporting test-time generalization to unseen object geometries and novel behavior triplets. The official sources read are the paper, the project page and a GitHub repository. No Hugging Face page was read for this entry, so none is linked. TACO is a hand-object research dataset, and the sources read do not describe it as robot data.

TACO is third-party research. Firsthand is not affiliated with its authors or their institutions, and makes no claim that TACO includes Firsthand data or hardware or that any model was trained on Firsthand data. Firsthand captures first-person human video, collected fast, on demand and to a written spec. It is not robot or teleoperation data. The fields the site states are head and wrist camera video, depth derived from a calibrated stereo rig, 3D hand pose at 21 keypoints per hand with a per-joint visibility flag, action segments, and 6-DoF IMUs at 200 Hz on one clock. The only language field is one task instruction per episode.

The abstract frames each sequence as a tool-action-object composition for daily human activities, with both hands in play. The README organises its files by <tool, action, object> triplet, and its visualization examples write triplets as (stir, spoon, bowl) and (brush, brush, box). The project page says the dataset supports generalization to unseen object geometries and to novel behavior triplets.

The counts depend on the source. The abstract says 2.5K motion sequences. The README describes a whole dataset, Version 1, of 2317 motion sequences spanning 151 triplets, with 206 high-resolution object models, and a pre-released version of 244 sequences spanning 137 triplets. The abstract and README figures differ, so this page treats neither as a total, and which version any experiment used is Not stated in the sources read. Total duration in hours and the number of subjects are Not stated in the sources read.

The abstract says TACO uses a fully automatic data acquisition pipeline that combines multi-view sensing with an optical motion capture system, designed to expand the data scale rapidly. Per the README, Version 1 contains egocentric RGB-D videos, 12-view allocentric RGB videos, camera parameters, hand-object pose and mesh annotations, automatic hand-object 2D segmentations, and marker-removed 12-view allocentric RGB videos. Each sequence folder holds left and right hand shape and pose files and object pose files for the tool and the target.

The README states that each egocentric depth image is a 1920×1080 uint16 array with a depth scale of 4000, so values are stored in units of 0.25 millimetre. It also says its hand files are visualized with MANO hand models, which a user downloads separately. How the egocentric camera is worn, the recording frame rate and the sensor models are Not stated in the sources read.

The abstract names three benchmark tasks: compositional action recognition, generalizable hand-object motion forecasting and cooperative grasp synthesis. The README supplies a train-test split file with a training set and four test sets: S1 with no generalization, S2 with geometry-level generalization, S3 with interaction-level generalization and S4 with compound generalization.

The abstract says the experiments reveal new insights, challenges and opportunities. The numerical results are Not stated in the sources read, and this page reports none.

All four are research datasets about hands and objects, but they record different things. The site’s ARCTIC entry covers two hands manipulating articulated objects, its OakInk2 entry covers bimanual manipulation in complex daily tasks, and its DexYCB entry covers hand grasping of objects. TACO’s distinguishing feature in the sources read is the tool-action-object triplet structure.

The comparison table takes TACO’s column from the sources read for this entry and the other three columns from their own entries on this site. The figures are not normalised, since each source counts sequences, frames and objects its own way, and TACO’s frame count is Not stated in the sources read. This page ranks none of them.

The README states that the work is licensed under a CC BY 4.0 license. Whether that covers every file, including the object models and any third-party material, is Not stated in the sources read. This page is not legal advice, so read the repository’s own licence text before a commercial program relies on the data. For contrast, the site’s ARCTIC entry reports its README as saying the data is not available for commercial use, and its DexYCB entry reports CC BY-NC 4.0 for the data.

The README links download locations for the pre-released version and the whole dataset, and lists a backup of each. Any access steps beyond those links are Not stated in the sources read.

TACO’s tools, objects and capture rig were fixed by its authors. When a program needs different tools, tasks, environments or people, write them into a spec. For two-handed work the site states that both hands stay in frame with per-hand 3D pose and visibility flags, including long continuous clips of fine bimanual work. If contact signals matter, ask for the separate pilot-ready tactile glove tier, time-synced to a head-mounted stereo camera by hardware timestamps and an IMU.

Confirm the spec on a Spec Pilot, a small fast first batch. Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks. Billing is per validated hour, and delivery is in the format your training stack already reads: RLDS, LeRobot, WebDataset or HDF5. Whether Firsthand delivers hand or object meshes like TACO’s is Not stated.

What is the TACO dataset?

A published bimanual hand-object-interaction dataset of tool-action-object compositions for daily human activities. The abstract reports 2.5K motion sequences with third-person and egocentric views, hand-object 3D meshes and action labels.

What does TACO stand for?

The paper title spells it out as Tool-ACtion-Object: TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understanding.

How many sequences does TACO have?

The abstract says 2.5K motion sequences. The README says Version 1 has 2317 sequences spanning 151 triplets, and a pre-released version has 244 sequences spanning 137 triplets. The figures differ, and the reason is Not stated in the sources read.

Can TACO be used commercially?

The README states that the work is licensed under a CC BY 4.0 license. Whether it covers every file is Not stated in the sources read, so read the repository’s licence text yourself. This is not legal advice.

Does TACO include language annotations?

Not stated in the sources read. The abstract names action labels, hand-object 3D meshes and paired third-person and egocentric views.

Does Firsthand use TACO, and can I request a custom dataset?

No. Firsthand is not affiliated with the TACO authors or their institutions, and makes no claim that TACO uses Firsthand data or that any model was trained on it. You can request a custom egocentric dataset: write the tasks, fields and format into a spec, confirm it on a Spec Pilot, and billing is per validated hour. The only language field is one task instruction per episode.