Guide
How to Evaluate a Robotics Data Vendor
Short answer
Evaluate a robotics data vendor with a four-step process: write a skill spec, request a small sample or Spec Pilot against it, score the delivered footage on a weighted scorecard (sync error, annotation depth, license, consent, reject policy, billing unit), and only then commit to volume. Never buy from a demo reel alone.
Last reviewed:
What is the actual due-diligence process?
Vendor evaluation for robotics training data is a four-step process, not a single decision. First, write a skill spec that names the tasks, environments and instruction phrasing your policy needs. Second, request a small sample against that exact spec — a free sample pack or a paid Spec Pilot, not a generic demo reel. Third, score what's delivered on a weighted scorecard against your own priorities, not the vendor's pitch deck. Fourth, only commit to a full custom-collection volume once the sample has passed.
What does a weighted scorecard look like?
| Scorecard row | Weight | What to look for |
|---|---|---|
| Sync error (measured, not described) | High | Ask for a number and method, not the word "aligned." |
| Annotation depth (pose, depth, action segments) | High | RGB and narrations alone are not manipulation-ready. |
| License & commercial rights | High | Confirm you can train and ship a commercial model on the output. |
| Consent chain, delivered per batch | High | A written release per participant should ship as an artefact, not a promise. |
| Reject policy and published reject rate | Medium | A vendor that shows you what it discarded is protecting your signal. |
| Billing unit (validated vs. raw hour) | Medium | Per-validated-hour billing aligns the vendor with your outcome. |
| Time to first batch, on demand | Medium | Ask for a realistic median, and an honest number for thin environments. |
| Coverage histogram published | Low | Fill rates by condition tell you where the data is thin before you buy. |
Vendor evaluation scorecard template
Weight each row by how much it actually affects your policy, score every candidate vendor against the same sample, and total the result. A vendor that scores well only on price or turnaround, and poorly on sync error or license, is not passing due diligence.
What should the sample request actually contain?
- Raw, unedited footage against your written spec, not a cherry-picked highlight reel
- The measured sync error and the method used to produce it, not a qualitative claim
- At least one failure case, transit, or occlusion clip, not only clean successful grasps
- The exact delivery format (RLDS, LeRobot, or WebDataset) you will actually train on
- Proof of per-participant written consent for the sample batch
What happens after the scorecard is filled in?
If a vendor's sample clears every high-weight row, a Spec Pilot, a small, fast first batch that proves the spec and the accept criteria, is the normal next step before scaling to full volume. If it fails a high-weight row, that is a disqualifying result regardless of price or speed; a vendor that cannot show a measured sync number or a clean commercial license is not offering data you can train and ship on.
FAQ
How to Evaluate a Robotics Data Vendor, answered.
01What is the difference between a scorecard and a checklist?
A checklist tells you what to look at. A scorecard makes you weight each item and score every vendor against the same sample, so the comparison is a number, not an impression.
02How many vendors should I put through this process?
Enough to have a real comparison, typically two or three, each scored against the identical written spec so the scorecard rows mean the same thing for every candidate.
03Is a demo reel ever enough to decide on?
No. A demo reel is cherry-picked by definition. Score the vendor on a sample delivered against your own spec, including the reject-worthy footage, not their best clips.
04What disqualifies a vendor outright?
No measured sync error or method, a non-commercial or ambiguous license for a commercial project, or no written per-participant consent are each disqualifying regardless of how the other rows score.
05How does this relate to choosing a provider in general?
This page is the process and scorecard mechanics; the companion guide on choosing an egocentric data provider covers the underlying criteria in more depth.
Check it against the sample pack.
40 episodes across 4 environments, delivered in the exact schema these guides describe.