What Is the DexYCB Dataset?

DexYCB is introduced in DexYCB: A Benchmark for Capturing Hand Grasping of Objects (arXiv 2104.04631), accepted to CVPR 2021, with an official project page and an official toolkit repository on GitHub. The abstract says the authors first compare DexYCB with a related dataset through cross-dataset evaluation, then benchmark state-of-the-art approaches on three tasks, and finally evaluate a new robotics-relevant task: generating safe robot grasps in human-to-robot object handover. The word benchmark matters here. DexYCB is a dataset together with fixed evaluation protocols, so different methods can be scored the same way. It is a hand-object grasping research resource. The sources read do not describe it as robot teleoperation data.

DexYCB is third-party research. Firsthand is not affiliated with its authors or their institutions, and makes no claim that DexYCB uses Firsthand data or hardware, or that any named model or dataset relates to Firsthand data. The site states custom egocentric collection to a written spec, fast and on demand, delivering first-person human video with 3D hand pose at 21 joints per hand and a per-joint visibility flag, calibrated stereo depth and 6-DoF IMUs at 200 Hz. The only language data tied to Firsthand capture is one task instruction per episode. Firsthand captures first-person human video, not robot or teleoperation data.

The official toolkit README describes each image sample as the paths to a colour image and a depth image, a label file, camera intrinsics, the IDs of the objects present, the index of the object being grasped, whether the hand is right or left, and the hand’s MANO shape parameters. The label file packs five annotations: a segmentation map, 6D object poses, the hand’s MANO pose, 3D hand joints and 2D hand joints.

In the segmentation map each pixel is background, one of the object classes (IDs 1 to 21 in the label schema) or hand. Object poses are stored per object as a rotation matrix and a translation. The hand pose is stored as MANO pose coefficients in a PCA representation plus a translation, the 3D joint array holds 21 joints in camera coordinates, and the 2D array holds the same joints in image space. The README says that where an image has no visible hand or the annotation does not exist, the hand pose is all zeros and the joint arrays are all -1. How many cameras recorded each sequence, the image resolution, the frame rate and the number of sequences are Not stated in the sources read.

The project page states that DexYCB is licensed under CC BY-NC 4.0. It offers two ways to download the data: one compressed archive that holds everything, listed at 119G, or ten per-subject archives of 12G each plus smaller archives for BOP-format data, camera calibration and object models. The extracted folder holds one directory per subject recording, a calibration folder and a models folder. The toolkit README links studio hardware and recording guides, which this page has not read.

The toolkit is a separate artefact with a separate licence: the GitHub repository lists GPL-3.0. It is a Python package of evaluation and visualisation tools, which its authors say is tested with Python 3.7 on Linux, and it needs the MANO model files from the MANO website. Because the data licence name includes NC, a programme with commercial use in mind should read the full terms on the project page itself before relying on the data.

The abstract names three benchmarked tasks and one new one. The three are 2D object and keypoint detection, 6D object pose estimation and 3D hand pose estimation. The new robotics-relevant task is generating safe robot grasps in human-to-robot object handover. The toolkit README maps these to four evaluations, called COCO, BOP, HPE and Grasp, each with its own results file format.

Per the README, detection results use the COCO results format. Object pose results are a CSV with one pose estimate per line, a rotation matrix and a translation in millimetres. Hand pose results are a text file with one line per image sample, holding the sample index and 63 numbers: the x, y and z of the 21 joints, in millimetres, in camera coordinates. To speed up the BOP evaluation, the README says only a set of subsampled keyframes is evaluated, flagged on each sample, and cites the supplementary paper. The Grasp evaluation takes the object pose and hand segmentation results, generates grasps for handover from them, and uses a set of 100 pre-generated grasps for each object when it evaluates those grasps. Benchmark scores are not reproduced here.

DexYCB gives researchers a shared yardstick for how well a method recovers where a hand and the object it holds are in space. The handover task is the robotics link: it asks for grasps a robot could use to take an object from a person, evaluated against pre-generated reference grasps.

Training robot policies on DexYCB is Not stated in the sources read, and this page draws no conclusion about how DexYCB would perform as pretraining data for any robot model.

DexYCB and custom egocentric capture answer different questions. DexYCB is a fixed public benchmark with object models and evaluation code, so results from different methods are comparable. Custom egocentric capture is a data service: paid, consented contributors recording first-person video to a written spec, with fields and format set in advance. This page makes no ranking between them.

The comparison table below sets only what the official sources state against what the site states, and where a source is silent the cell says so. Camera viewpoint is among the things the sources read do not state, so no claim is made about whether DexYCB is egocentric.

When a program needs grasps of objects, environments, tasks or people that a public benchmark does not cover, specified in writing, and the data has to be usable for the program’s own purpose. The site states that both hands stay in frame with per-hand 3D pose and visibility flags. Language beyond one task instruction per episode is not something the site states for Firsthand capture, and object 6D pose labels of the kind DexYCB provides are not among the fields this page lists for Firsthand, so a program that needs either should raise it when the spec is written.

Confirm the spec on a Spec Pilot, a small fast first batch. Where coverage is deep, the median from signed spec to first validated batch is 48 hours; where it is thin, three to four weeks. Billing is per validated hour, and delivery is in the format your training stack already reads: RLDS, LeRobot, WebDataset or HDF5.

What is the DexYCB dataset?

A published dataset and benchmark for capturing hand grasping of objects, from a CVPR 2021 paper. It benchmarks 2D object and keypoint detection, 6D object pose estimation and 3D hand pose estimation, and adds a human-to-robot handover grasp task.

What does DexYCB annotate?

Per the toolkit README, each sample has a segmentation map, 6D object poses, a MANO hand pose, 21 3D hand joints in camera coordinates and 21 2D hand joints. Camera count, resolution and frame rate are Not stated in the sources read.

Can DexYCB be used commercially?

The project page states the data is licensed CC BY-NC 4.0, and the toolkit repository lists GPL-3.0. Read the full terms on the project page before relying on the data for a commercial program.

Is DexYCB a robot dataset?

Not stated in the sources read. The abstract includes a robotics-relevant task, generating safe robot grasps in human-to-robot object handover, but does not describe robot teleoperation data.

Does Firsthand use DexYCB, or is it affiliated with its authors?

No. Firsthand is not affiliated with the DexYCB authors or their institutions, and makes no claim that DexYCB uses Firsthand data or hardware, or that any named model or dataset relates to Firsthand data.

Can I request a custom egocentric dataset?

Yes. Write the tasks, environments, fields and format into a spec, confirm it on a Spec Pilot, and billing is per validated hour. Firsthand captures first-person human video, not robot or teleoperation data, and the only language field is one task instruction per episode.