How Does Custom Robotics Data Collection Work?

Custom robotics data collection is a managed service that captures egocentric video and other robotics training data against a written specification, rather than selling footage that already exists. Instead of picking from a fixed catalog, a buyer describes a task and a coverage need, and a pipeline turns that need into data that clears explicit accept criteria.

Firsthand runs this as one accountable pipeline: the team that writes the spec is the team that ships the data, so the accept criteria and the consent chain are enforced at every stage rather than lost between a crowd platform, a labelling vendor, and a QA tool.

Concretely, that pipeline runs in seven stages: discovery, specification, sourcing, capture, annotation and QA, anonymization, and licensed delivery. Each of the sections below covers one or more of those stages in the order a buyer actually experiences them — starting with the written spec that everything downstream is measured against.

That single-owner structure is a deliberate design choice, not a marketing phrase. Sourcing, capture, annotation, QA, and licensing sit inside one accountable pipeline, so when something is wrong there is one team to call rather than a crowd platform blaming a labelling vendor blaming a tool. Consent is captured at sourcing and travels with the item all the way through delivery — a stitched-together pipeline of separate vendors is exactly where consent chains break, and an end-to-end one is where they hold.

The practical upside for a buyer is that a custom collection becomes data received, not a project to manage: no orchestrating separate vendors, no reconciling mismatched formats, no chasing who actually owns the resulting license. A single brief produces an owned, structured, documented dataset.

That single-owner structure also changes who is accountable when a batch is late or a coverage cell is thin. In a stitched-together pipeline, a buyer chasing an explanation is often bounced between the sourcing vendor, the labelling vendor, and whoever ran QA, none of whom controls the whole picture. In a single accountable pipeline, the same team that wrote the spec, ran capture, and delivered the data can explain a shortfall against the spec directly, because they own every stage that could have caused it.

The same pipeline serves a range of AI training needs beyond a single robot skill — embodied AI and robotics, computer vision, and other modalities that need task-specific egocentric or sensor data rather than generic web-scraped footage. What stays constant across those use cases is the structure: a written spec, a contributor pool matched to that spec, capture against explicit accept criteria, and a licensed, documented delivery. Only the modality, sensors, and annotation schema change from one engagement to the next.

That constancy is what "custom" actually means in this context. It does not mean bespoke process for every buyer — it means a fixed, repeatable process applied to a spec that is unique to the buyer’s task. The pipeline itself is standardized precisely so that the variable part, the spec, can change from engagement to engagement without the underlying quality and consent guarantees changing with it.

That standardization is also why a buyer new to commissioning data collection does not need in-house expertise in capture hardware, annotation schemas, or consent workflows to get a usable dataset out of the process. The spec stage exists precisely to translate a buyer’s task requirements — stated in the buyer’s own terms, as a skill, a condition mix, and an acceptance test — into the operational detail the pipeline needs to execute, so the buyer’s expertise in their own product does not have to be paired with expertise in robotics data collection for the engagement to work.

A skill spec is the written contract the whole collection is measured against. It names the exact task by verb and noun (not "cooking" but "dice an onion"), states the geometry that must stay in frame, budgets hours per condition cell — daylight, low light, cluttered, failure cases — and writes the reject rule and one acceptance test before capture starts.

This is also the stage where the population the task needs gets named explicitly — which languages, which age bands, which devices, which environments — rather than left implicit and discovered as a gap after delivery. Recruitment then draws on a global contributor pool spanning many countries and languages, screened against exactly those spec cells, so the population named in the spec is the population actually captured.

A finished spec is not a prerequisite to start: turning a problem into a written, testable spec with accept and reject criteria is itself an early stage of the pipeline, done together on a working call. You bring the failure the model has today; the spec gets written and handed back as a file you approve before anyone is briefed.

A signed spec functions as the single reference document for every later stage. Recruitment screens contributors against its named population; capture briefs contributors against its geometry and reject rule; QA measures delivered items against its acceptance test; and delivery packages the data according to its named format and schema. When a question comes up mid-collection — is this angle acceptable, does this lighting count as the low-light cell — the answer is looked up in the spec rather than decided fresh each time, which is what keeps a large batch consistent across many contributors and many days of capture.

That working call is discovery, and it starts from the failure rather than from a schema: what the model is being trained to do, and exactly where it breaks today. The output of discovery is a shared problem statement and a first read on feasibility, reach, and rough timeline — the raw material the spec stage then turns into modality, languages, demographics, devices, environments, volume, and edge-case coverage, plus the delivery format, metadata schema, and consent and licensing basis, all agreed up front and signed before capture starts.

See how to write a skill spec for the full five-part template — verb and noun, geometry, condition budget, reject rule, acceptance test.

Each of those five parts exists to close a specific failure mode. A vague verb and noun ("cooking" instead of "dice an onion") leaves capture free to drift toward whatever is convenient to film. An unstated geometry lets a contributor frame a shot so the hands or the object of interest fall out of view. A condition budget that only states a total hour count, with no breakdown by lighting or clutter, lets a batch quietly fill on the easy cells and skip the hard ones. A reject rule written after the fact turns every disagreement into a negotiation instead of a check against a pre-agreed line. And without a written acceptance test, there is no way to know a batch is done rather than merely delivered.

The acceptance test deserves its own emphasis because it is the part most often skipped in an informal brief. A spec can name the right verb, the right geometry, and a sensible condition budget, and still leave "done" undefined — which in practice means the buyer and the pipeline discover they disagreed about what counted as finished only after a batch has already shipped. Writing the acceptance test as a concrete, checkable statement (this many episodes, in these condition cells, each clearing the reject rule) before capture starts removes that disagreement from the end of the engagement and settles it at the beginning, where it is cheap to resolve.

A spec pilot (pilot data collection) is a small, fast first batch captured against the written spec to prove the spec and the accept criteria before committing to full volume. It surfaces the real reject rate, coverage fill, and sync numbers on real footage at low volume, rather than asking a buyer to commit to scale on a pitch deck.

A pilot is optional, not mandatory. No standing program is required to start a custom collection, and a focused brief is a normal starting point; many teams choose a spec pilot specifically to prove a new or unfamiliar spec before scaling volume.

The decision usually comes down to how much is already known about the population and the task. A buyer who has run a similar collection before, or whose spec covers a well-served population and a familiar task, often moves straight to full volume. A buyer testing an unfamiliar condition mix, a rare embodiment, or a spec they are not yet confident is complete gets more value from seeing real reject and fill numbers on a small batch before committing the full budget.

Once a pilot (or an already-familiar spec) clears the accept criteria, the identical pipeline runs at the volume the spec calls for: the same skill spec, the same accept and reject rules, the same billing unit. Nothing about the process changes between a small pilot and a full-volume program — only the hours captured do.

That consistency is the actual value of running a pilot first: a buyer scaling from a pilot to full volume is not taking on new process risk, because the process at scale is provably the same process that already passed at small volume. Nothing is re-negotiated, re-specified, or re-approved between the two — the spec, the accept criteria, and the billing unit all carry over unchanged.

Recruitment and staffing draw on the screened contributor pool matched to every cell of the spec — language, accent, age band, device, environment, expertise — and thin or under-served segments get a realistic staffing timeline rather than an optimistic one, agreed up front during specification.

Scaling volume up does not mean scaling scrutiny down. Every batch, large or small, is measured against the same signed accept and reject criteria from the spec stage, and every batch produces its own reject log. A high-volume program does not trade thoroughness for throughput; it runs the same per-batch check at every batch, however many batches the full volume requires.

This is also why a spec is written by condition cell rather than as one aggregate hour count. A policy fails on the conditions it never saw, not on the ones it saw a lot of, so budgeting hours per cell — daylight, low light, cluttered, failure cases — surfaces exactly where coverage is thin before training ever does. Fill rates are reported against that same document at every batch, so both the buyer and the pipeline are looking at the same coverage picture as volume scales up.

That per-batch reporting is what turns scale from a black box into something a buyer can steer mid-collection. If a batch report shows the low-light cell filling slower than daylight, that is visible before the whole volume is spent chasing an easy cell — the buyer and the pipeline can decide together whether to extend the timeline for the thin cell or accept a documented gap, rather than discovering the gap only once the finished dataset is already in hand.

Scale also means the pipeline is drawing from a genuinely global pool rather than a single regional labor market, which is what makes rare condition cells staffable at all. A spec that needs a specific language, a specific accent, a specific home layout, or a specific device is not searching a local pool for a coincidental match — it is matching against a pool that spans many countries by design, so a thin cell in one region can still be filled from contributors elsewhere who meet the spec’s criteria.

Contributors are briefed and given paid practice runs so they know what "good" looks like before the real take, then paid hourly for their time — including setup and retakes, never a per-item bounty. Specialized hardware is used where the project needs it, including a synchronized egocentric rig for multi-stream capture.

Paying hourly rather than per accepted item is a deliberate choice about who bears the risk of a difficult condition cell. A per-item bounty pushes a contributor to rush or to avoid the hard cells entirely, since only accepted items get paid; an hourly rate means a contributor is paid for the retake, the awkward lighting setup, and the honest attempt at a rare condition, and the pipeline — not the contributor — absorbs the cost of a reject.

Every sensor shares one PTP (IEEE 1588) clock domain with hardware-triggered exposures, so RGB, depth, IMU and pose all carry timestamps on the same timebase. The measured residual offset between streams runs a median of 1.12 ms and a p99 of 1.94 ms, against a 2.0 ms reject ceiling — sub-frame synchronization is what makes multi-stream egocentric data trainable rather than merely co-recorded.

That reject ceiling is enforced per episode, not audited after the whole batch is delivered. An episode whose measured sync residual exceeds 2.0 ms is discarded at capture rather than shipped and left for the buyer to discover during training — which is also why the reject log in the QA stage includes sync failures alongside spec-coverage failures.

Calibration follows the same discipline as sync: it is checked and logged per session rather than assumed to hold from a one-time setup. Sensor rigs drift with heat, handling, and time, so a calibration captured at the start of a long capture day is not guaranteed to still be accurate by the end of it — which is why calibration state, like sync residual, is something the pipeline records against every episode rather than treating as a fixed property of the hardware.

Sub-frame sync matters specifically for multi-stream egocentric data because a policy that fuses RGB, depth, IMU, and pose learns the relationship between those streams, not just each stream individually. If the timestamps drift even a few milliseconds apart, the fused representation is teaching the model a relationship between hand position and visual frame that did not actually hold at that instant — a subtle error that does not show up as a corrupted file, only as a policy that behaves worse than the raw footage would suggest it should.

Practice runs serve the same purpose as the written spec, but at the individual contributor level rather than the batch level. A spec can name the exact geometry and reject rule in writing, but a first-time contributor still benefits from seeing what a passing take actually looks like before the take that counts is recorded. Paying for that practice run, rather than treating it as unpaid overhead the contributor absorbs, is part of the same hourly-pay logic that covers retakes and setup — the pipeline pays for the learning curve so the contributor is not incentivized to skip it.

Raw capture is transcribed, labelled, keypointed, segmented, or event-tagged as the spec requires, structured into the buyer’s delivery schema rather than delivered as a folder of files to wrangle. A second reviewer covers a sampled or full pass depending on the annotation’s density and stakes.

Which annotation type applies depends on what the spec asked the data to carry: keypoints for a manipulation task where joint or hand position matters, segmentation masks where the boundary of an object or surface matters, event tags where the moment a sub-task starts and ends matters, and transcription where the spoken or narrated content of the capture matters. The spec’s delivery schema determines this up front, so annotation is not guessed at during the QA stage.

Every item is then measured against the signed accept and reject criteria from the spec stage, not a vibe check. Items that miss are rejected and recaptured at no extra cost, and the reject log — with reasons, not just pass counts — is visible to the buyer.

Billing follows the same discipline: the unit is a validated hour, meaning footage that has already cleared the spec’s accept criteria, not a raw hour of capture before anyone has checked it. A buyer is not paying for a shelf of footage that might work; they are paying for hours that already have.

A sampled review pass and a full review pass are not interchangeable, and the spec is what decides which one applies. High-stakes or low-volume annotation — a rare failure condition, or a small batch where a single missed keypoint changes the whole batch’s reliability — gets a full second pass. Dense, high-volume annotation on well-understood conditions can be sampled, because a statistically sound sample catches a systematic labelling error just as reliably as reviewing every item, at a fraction of the review cost.

Every contributor signs a release before capture, and the same standard covers anyone who appears in frame incidentally — family members, coworkers, bystanders — not only the person performing the task under contract. Faces, license plates, and screens are detected, tracked, and irreversibly blurred, with a human reviewer confirming the result rather than trusting automated detection alone.

The output is a validated set plus a reject log and an anonymization record a buyer can audit, alongside the signed consent releases that make up the provenance behind the dataset. This is what lets a buyer show their own counsel, months after delivery, exactly which release covers which piece of footage — rather than reconstructing that trail after the fact from a vendor who may no longer be reachable.

Consent and anonymization are handled as two separate, sequential checks rather than one combined step, because they answer different questions. Consent asks whether a person agreed to be recorded and to how the footage will be used; anonymization asks whether anyone who did not consent — a bystander, a family member walking through frame — is still identifiable in the delivered footage. A clip can pass one check and fail the other, which is exactly why both are checked independently rather than assuming one implies the other.

Delivery ships in the format specified during the spec stage — RLDS, LeRobot, WebDataset, HDF5, zarr, Rerun, WAV, JSONL, and others — with the per-item metadata, the signed consent releases, the anonymization record, and a data card recording jurisdictions and provenance, all under a perpetual, buyer-owned commercial license rather than a shared or revocable one. Where a buyer needs a dataset collected only for them, exclusivity is available at this stage rather than the same footage being resold to competitors.

The point of shipping in the buyer’s own format and schema, rather than a generic export, is that the dataset is meant to load directly into an existing training pipeline. A buyer should not need to write a bespoke parser or reconcile a mismatched schema just to start using data they specified in the first place.

Format flexibility matters because robotics and embodied-AI teams do not converge on one standard the way, say, image classification did. A team training on RLDS or LeRobot pipelines needs episodic data with matching action and observation streams; a team validating with Rerun needs a visualizable trace; a team running classical perception baselines may just need HDF5 or zarr arrays with no policy-specific structure at all. Naming the target format in the spec stage, before capture starts, avoids a costly reformatting pass after delivery.

A working session on the model, task, and failure conditions can start without a finished spec — turning the problem into a written spec with accept and reject criteria is an early pipeline stage, not a prerequisite. From there, the same seven-stage pipeline runs: contributor recruiting, capture, per-batch QA, anonymization, and licensed delivery, with rejected items recaptured at no extra cost.

How quickly staffing completes depends on how common or rare the population is: common languages, devices, and environments staff quickly, while thin or under-served segments get a realistic timeline agreed during specification rather than an optimistic one.

What keeps a custom collection from stalling is that the same seven stages run whether the engagement is a one-off brief or an ongoing program: discovery and specification are not a separate sales process bolted onto capture, they are stage one and two of the identical pipeline that ends in licensed delivery.

This is also why the discovery call matters for speed, not only for accuracy. A buyer who arrives with a clearly demonstrated failure — the model doing the wrong thing on camera, or a specific condition it has never seen — gives the spec stage a concrete target to write against, which shortens the back-and-forth needed to reach a signed spec. A buyer who arrives with only a general goal spends more of the early pipeline turning that goal into something testable, which is time well spent, but time nonetheless.

An off-the-shelf dataset is fixed: the coverage, license terms, and consent basis were set by someone else’s spec, for someone else’s task. Custom collection is worth it when the exact task, condition mix, or embodiment the buyer needs is not covered by what already exists, or when the license and consent terms of an existing dataset do not fit the buyer’s commercial use.

The trade-off runs the other way too. An existing dataset that already covers the task, embodiment, and condition mix a buyer needs, under a license that already fits, is available immediately with no spec-writing or staffing timeline at all — custom collection exists for the gap an off-the-shelf catalog cannot fill, not as a default first step.

In practice, the two are not always an either-or choice. A buyer can start with an off-the-shelf dataset for broad coverage and commission a custom collection only for the specific condition cells or failure modes that dataset does not cover — the rare lighting condition, the unusual embodiment, the edge case that shows up as a production failure but never as a labelled example. Deciding which parts of a coverage gap are worth closing with a custom spec, versus accepted as a known limitation, is itself part of what the discovery call is for.

Cost and timeline are the practical constraints that usually settle the question once coverage and licensing are equal. An off-the-shelf dataset that already fits is cheaper and faster than any custom spec, because there is no pilot to run and no batch to staff — the only reason to pay for a custom collection is that the fit off-the-shelf is not close enough to justify the discount.

The comparison guide below walks through this trade-off in more depth, covering coverage, licensing, exclusivity, cost and timeline side by side, and is the right next read for a team still deciding which path fits their situation before committing to either one.

None of this makes off-the-shelf the wrong default. For a broad, common skill — the kind of grasp, navigate, or manipulate task that many teams need and many catalogs already cover — an existing dataset is usually the faster and cheaper starting point, and custom collection is reserved for the specific gap that starting point leaves behind once a team has actually tried training against it and found where it falls short.

The clearest signal that a gap is worth closing with a custom spec, rather than accepted as a known limitation, is a repeated production failure that traces back to a condition an existing dataset simply does not contain — not a hypothetical edge case, but one the team has actually observed. A spec written against an observed failure is easier to write precisely, easier to validate with a pilot, and easier to justify against the cost of a custom collection than a spec written speculatively against a condition that has not yet caused a real problem.

That is also why the pilot stage matters even when a buyer is confident the spec is right: it turns a speculative gap into an observed one before the full batch is staffed, so the decision to scale is based on real footage against the real acceptance test rather than on the spec document alone.

Taken together, the stages described above — spec, pilot, scale, QA, and delivery — are the same process whether the gap being closed is a single missing condition cell or an entirely new skill. What changes between engagements is the content of the spec and the size of the batch, not the pipeline that turns a spec into a delivered, licensed dataset.

A buyer evaluating whether custom collection is the right next step does not need to reach a final decision before the first conversation. The spec discussion itself is where coverage gaps, licensing needs, and timeline constraints get made concrete enough to compare against an off-the-shelf alternative — which is why starting that conversation costs nothing and commits to nothing beyond a pilot, and why it is the fastest way to find out whether a gap is worth closing with a custom spec at all.

The related guides linked throughout this page go deeper on each individual stage — how to write a skill spec, how a pilot is evaluated, how quality assurance is applied, how fast a collection can start — for a team that wants the detail behind any one step before moving forward with the process as a whole.

Read together, those guides and this one describe the same pipeline from two angles: this page as the end-to-end walkthrough, and each linked guide as the detailed reference for the stage it covers, so a buyer can move between the overview and the specifics without losing track of how any one stage fits into the whole engagement.

The stage-by-stage table and deliverables table above are meant to be used the same way: as a quick reference to return to once the surrounding prose has answered the underlying question, rather than as a substitute for reading the reasoning behind each stage the first time through.

The next step from here is the same regardless of which stage a buyer is trying to understand: request a custom dataset, describe the gap in the skill spec conversation, and let the pilot decide whether the collection is worth scaling before any full batch is committed.

License terms are frequently the deciding factor even when coverage looks close enough on paper. A dataset licensed for research use only, or shared non-exclusively across many buyers, can still be the wrong choice for a team shipping a commercial product — not because the footage is wrong, but because the buyer cannot actually use it the way they need to. A custom collection’s buyer-owned, perpetual commercial license, agreed at the spec stage rather than discovered in a catalog’s fine print, removes that risk before capture ever starts.

Exclusivity is a related but distinct question from license type. A buyer-owned commercial license means the buyer can use the delivered data without restriction; exclusivity, where available, additionally means the same footage is not also being sold to a competitor. A team building a defensible model on a narrow, hard-won skill often values exclusivity specifically because an off-the-shelf catalog dataset — even a well-licensed one — is available to any buyer, including a direct competitor, which is exactly the scenario a custom, exclusive collection is built to avoid.

See custom vs off-the-shelf robotics datasets for a fuller comparison of when each is the better fit.