What Is the OakInk2 Dataset?

OakInk2 is introduced in OakInk2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completion (arXiv 2403.19417), with an official project page and an official GitHub toolkit repository. The authors build on the earlier OakInk dataset, which they say lacks human demonstrations that cover the process of fulfilling object affordances, and lacks complex manipulation tasks that involve more than one affordance. OakInk2 takes an object-centric view with three levels. An Affordance is a function of an object or object part, such as a bottle cap securing the contents. A Primitive Task is the minimal hand-object interaction that fulfils an affordance. A Complex Task combines Primitives that depend on one another, and the paper represents it as a directed acyclic graph called a Primitive Dependency Graph. On top of the dataset the authors explore Complex Task Completion: language models decompose a task objective into a sequence of Primitives, and a Motion Fulfillment Model generates bimanual hand motion for each one. OakInk2 is a hand-object research dataset. The sources read do not describe it as robot data.

OakInk2 is third-party research. Firsthand is not affiliated with its authors or their institutions, and makes no claim that OakInk2 uses Firsthand data or hardware or that any model was trained on Firsthand data. The site states custom egocentric collection to a written spec, fast and on demand, delivering first-person human video with 3D hand pose at 21 joints per hand and a per-joint visibility flag, calibrated stereo depth and 6-DoF IMUs at 200 Hz. The only language data tied to Firsthand capture is one task instruction per episode. Firsthand captures first-person human video, not robot or teleoperation data.

Per the paper, the capture platform pairs an optical motion-capture system of 12 infrared cameras, which tracks surface markers on the subject’s upper body, both hands and the interacting objects, with 4 commodity RGB cameras: 3 allocentric views and 1 egocentric view. All sensors are synchronised at 30 fps and calibrated against each other, and the videos are 848 × 480. The authors set up four manipulation scenarios: a kitchen table, a study room table, a demo chemistry lab and a bathroom table.

The paper reports 627 sequences of bimanual manipulation, 363 of them for Primitives and 264 for Complex Tasks, totalling 4.01M image frames. It lists 75 objects, 9 subjects and 39 affordances that map to 60 types of Primitive. The 38 long-horizon goals instantiate into 150 Complex Tasks. Total recording duration in hours is Not stated in the sources read.

The paper splits the annotations in two. The first is 3D motion: pose and shape for the upper body, hands and objects, with articulation parameters for articulated objects. Object poses come from the motion-capture markers, the body is fitted with the SMPL-X body model, and hand representations such as MANO are derived from that fit. The second is task specification: object affordances, the Primitives that fulfil them, Complex Tasks with goals and initial conditions, Primitive Dependency Graphs, expert commentary and the subject’s completion sequence.

The official README describes the released files. Frames sit in one folder per camera. An annotation file holds camera intrinsics and extrinsics, object transformations, and raw SMPL-X and MANO parameters. Program files give each Primitive’s time interval per hand, its objects, an interaction mode (left-hand main, right-hand main or both hands main) and a textual description. The project page lists text descriptions of task targets, Primitives and the subjects’ execution paths, and states that OakInk2 is currently in preview and that annotations are subject to future changes.

The abstract names interaction reconstruction and motion synthesis as the supported applications. The paper benchmarks hand mesh reconstruction in single-view and multi-view settings, and introduces Task-aware Motion Fulfillment, which generates hand motion for a Primitive from its textual description and the object trajectories. Together with the language-model decomposition of tasks into Primitives, these form the Complex Task Completion framework.

Training robot policies on OakInk2 is Not stated in the sources read. This page draws no conclusion about how OakInk2 would perform as pretraining data for any robot model. What it shows is how structured bimanual task data is organised: a task, its ordered sub-units and a pose for every hand in every frame.

OakInk2 and custom egocentric capture answer different questions. OakInk2 is a laboratory motion-capture dataset with marker-based ground truth, a fixed object repository and a rich, author-defined task hierarchy. Custom egocentric capture is a data service: paid, consented contributors recording first-person video to a written spec, with fields and format set in advance. This page makes no ranking between them.

The comparison table below sets only what the official sources state against what the site states, and where a source is silent the cell says so. Licence terms for OakInk2 are Not stated in the sources read, so a team with a commercial program should read the dataset terms on the project page and Hugging Face itself before relying on it.

When a program needs tasks, objects, environments or people that an existing dataset does not cover, specified in writing, and the data has to be usable for the program’s own purpose. For two-handed work the site states that both hands stay in frame with per-hand 3D pose and visibility flags, including long continuous clips of fine bimanual work. Language beyond one task instruction per episode is not something the site states for Firsthand capture, so a program that needs per-step text should plan for that separately.

Confirm the spec on a Spec Pilot, a small fast first batch. Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks. Billing is per validated hour, and delivery is in the format your training stack already reads: RLDS, LeRobot, WebDataset or HDF5.

What is the OakInk2 dataset?

A published dataset of bimanual hand-object manipulation for complex daily tasks, with multi-view video and 3D body, hand and object pose annotations, organised into Affordance, Primitive Task and Complex Task levels.

Does OakInk2 include language annotations?

Yes, from its authors: the project page lists text descriptions of task targets and Primitives, and the paper describes expert commentary. This is separate from Firsthand, whose only language data is one task instruction per episode.

Is OakInk2 a robot dataset?

Not stated in the sources read. The paper frames it around interaction reconstruction, motion synthesis and hand motion generation from task descriptions, not robot policy training.

Can OakInk2 be used commercially?

Licence terms are Not stated in the sources read. Check the terms on the official project page and Hugging Face before relying on it, and note the project page says the dataset is in preview.

Does Firsthand use OakInk2, or is it affiliated with its authors?

No. Firsthand is not affiliated with the OakInk2 authors or their institutions, and makes no claim that OakInk2 uses Firsthand data or hardware or that any model was trained on Firsthand data.

Can I request a custom egocentric dataset?

Yes. Write the tasks, environments, fields and format into a spec, confirm it on a Spec Pilot, and billing is per validated hour. Firsthand captures first-person human video, not robot or teleoperation data.