Egocentric Visual Inspection Task Data

Inspection is a loop of looking, handling and deciding. A person picks up a part, turns it under a light, brings it closer, compares it with a standard and then keeps it, sets it aside or rejects it. The task catalog describes that loop in plain terms: hold products under the inspection light, spot defects and pull them from the line. Every step is a physical action that changes what the inspector can see.

A first-person camera records that loop from the viewpoint a robot policy would act from. The inspector controls the view by moving the head and tilting the part, so the video shows not only what was looked at but how the view was changed to look at it. Hand pose and contact events record the handling that produced that view, and action segments mark where looking ends and handling or deciding begins.

It is worth being exact about what this data is. It records a person performing an inspection: the views, the handling and the sequence of steps. It is not a library of defects. The site does not state defect labels, a defect taxonomy, pass or fail outcome labels or measurement values as standard deliverables, and this page does not promise them. A buyer who needs any of them defines them in the written spec before capture begins.

The task catalog already lists inspection work across many environments. On production lines it includes quality inspection on a production line (inspect units on the line for defects, measure critical dimensions and reject failures) and in-line quality inspection (check parts on the line against the spec, measure and set aside defective units). In a food and beverage plant it includes inspecting and removing defective products under an inspection light, and a quality assurance and foreign-object inspection task on the belt.

In workshops, the catalog lists weld inspection and cleanup, where welds are inspected for defects, seams are ground and cleaned and parts are marked for rework, and component soldering, which ends with inspecting the joints. Returns work is full of inspection too: damaged-goods sorting, inspecting and sorting damaged goods with a handheld scanner and returns bins, returns processing in a warehouse, and processing a return on a retail floor set.

Service and facilities checks round out the list: vehicle diagnostics, inspecting and replacing air and cabin filters, resetting a hotel room after checkout and checking every item against the standard, inspecting and sorting fresh strawberries at a produce packing facility, and package inspection and bag search at a mock security checkpoint. These tasks differ in what is inspected, but share the same first-person structure of look, handle, rotate, check and decide.

A custom spec is scoped from that same vocabulary. You describe the items, stations, standards and decisions your policy needs, and the written spec covers them, with the catalog tasks as a starting point rather than a fixed menu.

The catalog places inspection tasks in a plastic and light-assembly manufacturer, a food and beverage manufacturing plant, a metal fabrication and welding workshop, an electronics manufacturing environment, the back of house of a retail store, a warehouse and distribution center, a hotel, an auto service center, a produce packing facility and a mock security checkpoint. Each task row carries a status that says how soon capture can begin.

The site defines those statuses plainly. Ready to deploy means contributors and site access are set up and capture starts once a spec is signed. Active collection means recording is running now, so a spec can join an existing capture stream. Available on request means the work is scoped and recruited to spec when a buyer asks for it. In the current catalog the production-line quality inspection rows, the food plant inspection rows, weld inspection, damaged-goods sorting and room turnover are marked Ready to deploy, while component soldering, the retail return, the produce sorting row and the checkpoint rows are Available on request.

The site does not say that any of this capture takes place inside a particular customer factory, and this page does not claim it. The catalog names environment types, and where a collection happens is agreed in the spec.

Manufacturing workflow data captures a whole production procedure: a sequence of fastening, measuring, fitting and inspection steps run in a defined order against a written skill spec. Inspection appears there as one step in a longer chain. This page takes the opposite cut. It treats inspection as a task family in its own right, so the spec, the examples and the labels are organized around the look, handle, rotate, check and decide loop rather than around a production sequence.

Assembly task data is about fastening, measuring, press-fit and wire work at a bench. Workplace task data is a broad page covering many on-the-job tasks across industries. Data labeling and quality verification is about checking the labels and the batches Firsthand delivers, not about people inspecting parts. Each of those pages is a neighbour of this one, and the table below sets them side by side.

The distinction matters for how you write a spec. If your policy has to follow a whole procedure, the workflow page is the better starting point. If it has to learn to examine, handle and judge individual items, this page is.

The catalog names two capture modes. In the first, a head-mounted phone records the first-person view of real work on site, and a multiview rig adds synchronized third-person angles when the spec asks for them. In the second, a staged working set is recorded by a multiview rig while the operator performs the task with handheld grippers, so demonstrations map onto a robot end effector.

A head-mounted camera keeps the part, the hands and the station in frame from the viewpoint a manipulation policy will see from, with a wrist-mounted camera available alongside it for close-range detail when that matters more than the full-arm view. Depth, 3D hand pose, contact events and action segments are aligned to video on one hardware clock, synced under 2 ms. Distance-invalid pixels beyond the depth sensor reliable range are flagged rather than interpolated.

The site does not state a camera count, resolution, frame rate or lighting setup specific to inspection, so those are agreed in the written spec rather than assumed. That includes any dedicated inspection light: the catalog task mentions one, but the site does not describe how it is lit or measured.

Each episode follows the same episode schema used across Firsthand custom collections: ten streams synced under 2 ms, with head and wrist RGB, depth, 3D hand pose with per-joint visibility, contact and grasp events, and frame-accurate action segments. Warehouse collections also state instance segmentation alongside action segments and grasp and contact events; the site does not state it specifically for inspection tasks.

For inspection, hand pose tracks the operator through the pick-up, rotate, tilt and set-down motions, contact events mark when fingers make and break contact with the part, and action segments mark the start and end of each step, such as looking, handling and sorting. Because the labels share one clock with the video and depth, a step boundary lines up with the exact frame where it happens.

Object labels use the terms your spec sets: the skill spec decides which object classes are in scope before capture, so labels follow the vocabulary your training stack expects rather than a generic taxonomy.

Defect labels, a defect taxonomy, a pass or fail outcome label for each inspected item, measurement values such as dimensions or readings, and part, serial or lot identity are not stated anywhere on the site as standard labels. Some catalog task names mention spotting defects, measuring critical dimensions or rejecting failures, but those describe what the operator does, not a label delivered with the episode.

A per-episode success or failure label is also not stated. If your policy needs any of these, they are defined in the written spec as annotation depth, and the quote covers them before capture begins. The page does not promise them by default.

Every episode links to a signed contributor release through a documented consent chain, and the consent record is part of the delivery. Contributors are the people who perform the inspection tasks, so their release covers the footage captured of their hands and workspace.

Faces, plates and screens are detected and irreversibly blurred at ingest, and a reviewer confirms the result before anything ships. The anonymization record is delivered with the data, so a buyer can see what was blurred and when. That matters at an inspection station, where coworkers, monitors and paperwork can enter the frame.

Delivery formats are RLDS, LeRobot and WebDataset, plus HDF5, zarr and Rerun .rrd. The first three are the training-loader formats; HDF5 and zarr carry arrays for custom pipelines, and Rerun .rrd gives a multi-stream synchronized playback file for review, which suits visually checking an inspection episode before writing a loader.

Converter source is readable, so you can retarget the schema to your own loader, and the episode schema page lists the fields. Inspection episodes use the same schema as every other Firsthand collection, so a loader that already reads another Firsthand dataset reads an inspection dataset without changes.

You describe the items, stations, standards and decisions your policy needs. We turn that into a written spec covering tasks, environments, capture tier, annotation depth and accept criteria, then quote reach, timeline and price against that document before any capture begins.

The collection runs through the same seven-stage end-to-end custom collection pipeline as every Firsthand program: contributor recruiting, capture, per-batch QA, anonymization and licensed delivery. Quality is measured against signed accept criteria, billing is per validated hour, a reject log ships with every batch, and rejected items are recaptured at our cost.

Where coverage is already deep, the first validated batch lands in a median of 48 hours from signed spec. Thin environments typically take three to four weeks, because vetting operators there is the bottleneck, and we quote that timeline up front. The catalog status on each task row tells you which case you are in.

On-demand top-ups run against the same spec and accept criteria after each training run, so new inspection data stays consistent with what you already hold. A Spec Pilot, a small and fast first batch, can prove the spec and accept criteria before you scale volume.

Every custom collection ships under a perpetual, buyer-owned commercial license, and exclusivity is available so the data is never resold. The license is documented alongside the consent chain, so a policy trained on the data can go into a product.