Custom Data Collection vs Data Labeling Services

Custom data collection commissions footage that does not exist yet. It starts with a written skill spec — the task, objects, environment and acceptance test a batch has to clear — then recruits and pays contributors, captures against that spec, runs QA, and delivers the result on a buyer-owned license. Nothing is produced until contributors record it.

A data labeling service, by contrast, works on footage that already exists. The input is video or images the buyer (or the labeler, on the buyer’s behalf) already has; the output is structure added on top of it — bounding boxes, polygon outlines, per-pixel segmentation masks, object classes, tracked instances or keypoints. The labeling vendor is not the party that recorded the original footage or obtained the contributor’s consent to use it.

The two categories answer different questions. Custom collection answers “we don’t have footage of this task, environment or population”. Labeling answers “we have the footage, we need it structured for training.” A team can need one, the other, or both in the same program, depending on what already exists.

A custom collection engagement runs a written spec through a pipeline: a signed skill spec defining the task, environment, capture tier and accept/reject criteria; a Spec Pilot, a small, fast first batch that proves the spec and QA bar hold on real footage; scaled capture against that same pipeline in scheduled batches; and delivery under a buyer-owned license.

On Firsthand, the captured streams are synchronized first-person video — head and two wrist cameras — alongside active-stereo depth and IMU, every sensor hardware-triggered off one PTP clock domain with a documented sync ceiling. Calibration is solved and re-verified at the start and end of each session, with the drift between the two solves written into episode metadata. See /custom-collection/enterprise-workflow-video-data and /custom-collection/egocentric-data-collection-services for how the pipeline is structured stage by stage.

Because the labels and the footage are produced by the same pipeline on the same clock, annotation that depends on geometry — depth-aligned segmentation masks, metric 3D hand pose, contact and grasp events — is included as part of the collection rather than added afterward by a separate vendor working from an export.

A data labeling service takes footage as an input and returns structured annotations: bounding boxes (the smallest rectangle around an object), polygon outlines, per-pixel instance or semantic segmentation masks, object classes, instances tracked across frames, and keypoints for pose. See /custom-collection/image-and-video-annotation and /custom-collection/keypoint-pose-annotation for the full set of label types this site’s own annotation offering covers.

Firsthand’s annotation service is bundled with its own capture rather than offered as a standalone pass on footage a buyer already has elsewhere. It explicitly annotates video captured to a buyer’s spec on Firsthand’s synchronized hardware — a deliberate constraint, not a tooling limitation, because depth-aligned masks, 3D hand pose and contact events depend on the same clock and calibration the footage was recorded on, and those do not exist for footage shot on unsynchronized cameras. If your footage already exists on a different camera setup without synchronized depth and calibration, a generic labeling vendor working from 2D frames is the category that applies, and it will be limited to label types that do not require metric geometry — 2D boxes, polygons and 2D keypoints rather than depth-aligned masks or 3D pose.

New capture is the right call whenever the footage itself does not exist: a skill, environment or population your model is missing, where no existing recording — in-house or off-the-shelf — covers the condition. Labeling cannot manufacture footage of a task that was never recorded.

New capture is also the right call when the label types you need depend on synchronized sensing that the existing footage does not carry — depth-aligned segmentation, metric 3D hand pose, or contact events tied to a hardware clock. These require the capture and the annotation to share the same calibration and timing, which a labeling pass applied after the fact to unsynchronized footage cannot retrofit.

Labeling on existing footage is enough when the recording itself already covers what the model needs — the right task, environment and population are already on tape — and the gap is purely structural: the footage has no bounding boxes, no segmentation masks, no object classes, or no tracked instances yet.

It is also enough when the label types you need do not depend on synchronized depth or a hardware clock — 2D boxes, polygon outlines, or 2D keypoints can be produced from ordinary video without the camera rig that produced it having been calibrated to a PTP domain. The moment a deliverable needs metric 3D geometry tied to the footage’s own depth and timing, the constraint described above applies.

In a custom collection engagement, consent is built into the pipeline from the start: contributors are sourced, paid and signed, with releases stored per item and per jurisdiction as part of the service. Anonymization — detecting, tracking and blurring faces, plates and screens — runs before anything leaves QA, with a reviewer confirming the result, and the pass fails closed, meaning anything a reviewer cannot confirm is fully covered is held or discarded rather than shipped. The delivered data ships on a perpetual, buyer-owned commercial license with a data card. See /guides/buyer-owned-training-data for what that license structure covers.

A data labeling service working on footage it did not capture is not the party that obtained the original contributor’s consent. Whatever consent and rights apply to that footage depend entirely on how and where it was originally recorded; a labeling vendor can only add structure to the pixels it is given, not retroactively supply a release that was never collected. Buyers evaluating a labeling-only vendor need to confirm separately that the footage they are supplying was lawfully captured and that they hold the rights to have it labeled and used for training.

QA in a custom collection engagement checks two things: whether the captured footage itself meets the spec (the right task, the right geometry in frame, sync and calibration within tolerance) and whether the labels applied to it are correct. Both are checked against a written acceptance test from the skill spec, with a second reviewer covering a sampled or full pass depending on the label’s density and stakes. Items that fail are rejected, logged with a reason, and recaptured at the provider’s cost — not billed. Calibration error is recorded per episode, held below a documented reprojection threshold, so geometric soundness is checked rather than assumed. See /custom-collection/data-labeling-quality-verification for the full review workflow.

A labeling-only service, working on footage it did not capture, can only QA the labels it applies — whether a box, mask or keypoint is placed correctly. It has no basis to check or re-shoot the underlying footage if the footage itself has a problem (poor framing, missing geometry, an undocumented bystander), because it did not control the capture. That is a structural difference between the two categories, not a quality gap in any particular vendor: QA on a label-only engagement is bounded by what the footage already contains.

Yes, and on Firsthand this is the default rather than a special case. A custom collection engagement includes annotation on the footage it captures as part of the same pipeline: the skill spec fixes the object vocabulary and which label types each episode needs before capture starts, and the delivered episode carries both the raw synchronized streams and the labels together, under one data card and one license. See /custom-collection/egocentric-data-collection-services for how spec, capture, annotation and QA run as one managed program.

What this site does not offer is the inverse combination — labeling applied to footage a buyer already has that was not captured on Firsthand’s synchronized hardware. For that scenario, the practical path is either a generic labeling vendor for the 2D label types that do not require synchronized depth, or a new custom collection engagement if the gap is footage, not just structure.