Egocentric Data Collection Checklist

A collection checklist is an ordered set of items that you can check off before, during and after a first-person capture program. Each item is a claim a collection partner should be able to answer directly and in writing, such as which streams are captured, what the sync ceiling is, who signed a release, and what makes an episode fail. The value is not the list itself. It is that every item points at something that is either written into the spec or produced by the pipeline, so nothing is left to be discovered after delivery.

This page orders the checks by program stage rather than by dataset signal. That is the difference from the other checklist and methodology pages on this site, and it is why it exists as a separate page. The robotics dataset quality checklist is run by a buyer against a finished sample pack or delivered batch. The methodology guide describes how a collection is carried out. The scoping guide covers the five buyer decisions that go into a brief. This checklist sits across all of them: it follows a custom program from the first spec call to the delivered batch and gives you checkbox items and partner questions at each stage.

Related pages: the robotics dataset quality checklist (buyer-side check of delivered data), how to collect egocentric video data (methodology), and how to scope a custom robotics data collection (the five scoping decisions). This page does not repeat them; it sequences them.

The program runs in six stages. Each stage produces something you can ask to see, and that output is the input to the next stage. If a stage cannot produce its output, treat that as a gap in the program rather than an assumption in your favor.

Run the stages in order for a first program. For a repeat order against an approved spec, stages 1 to 3 are largely reused, and the work shifts to stages 4 to 6 against the same accept criteria.

The spec is the contract that every later stage is checked against, so it comes first. You do not need a finished one to begin. Firsthand drafts the skill spec with you on a working call, starting from the failure your model has today, and hands it back as a file you approve before anyone is briefed. A buyer who arrives with a demonstrated failure gives the spec a concrete target.

Name the exact task by verb and noun, not a category: "dice an onion", not "cooking". State the geometry that must stay in frame, such as both hands at grasp and the object throughout the episode. Budget conditions by cell rather than as one total, because a policy fails on the conditions it never saw and a per-cell budget shows where coverage is thin before training does. Name the population the task needs, including languages, age bands, devices and environments, since the population named in the spec is the population recruited.

• The skill is named by verb and noun, with the geometry that must stay in frame.
• A condition budget lists hours per cell, including failure cases, not one total.
• The population is named: languages, age bands, devices and environments.
• Volume is stated in validated hours, which is also the billing unit.
• The labels the data must carry are named, for example hand pose, contact events or action segments.
• The delivery format and schema are named before capture starts.
• The consent and license basis is agreed, including whether exclusivity is required.
• A reject rule and one acceptance test are written into the spec, and you approved the file.

On volume, this site suggests 50 to 100 validated hours per skill as a starting point, concentrated on the conditions the policy will actually meet, then scaling only where held-out evaluation shows the policy is still thin. Ask for hours where evaluation is weakest first rather than an even spread.

Ask a collection partner: Will you draft the spec with us and return it as a file we approve? Where are the reject rule and acceptance test recorded? Is volume measured in validated hours or raw recording hours? If the answer to the last question is raw hours, the count overstates what you can train on.

The rig determines which streams exist and whether they agree with each other in time. The instrumented rig stated on this site is a head-mounted global-shutter 4K/60 camera with a 120° lens, two wrist-mounted 1080p/60 cameras, an active-stereo depth sensor at 848×480/30, and 200 Hz 6-DoF IMUs. Not every order needs every stream. Depth, IMU and hand pose are added where the written spec calls for them, and a buyer who needs only RGB and action labels can scope to that.

The check that matters most for multi-stream data is synchronization. All streams are hardware-triggered onto a single PTP (IEEE 1588) clock domain, with per-stream nanosecond timestamps in that domain so the alignment can be verified independently. The stated residual is a median of 1.12 ms and a p99 of 1.94 ms against a 2.0 ms reject ceiling. An episode that exceeds the ceiling is discarded at capture rather than shipped.

Calibration is the second rig check. Each session is calibrated twice, once before capture and once after it ends. The opening solve uses a checkerboard target across 40 poses spanning every camera frustum, with per-camera intrinsics solved independently to a reprojection error under 0.28 pixels, head, wrist and depth extrinsics solved against the shared target and expressed in the head-camera frame, and IMU bias and scale estimated from a static hold plus a figure-eight excitation. The closing solve repeats the procedure, and the drift between the two is recorded so a change mid-session shows up as a number rather than an assumption. Intrinsics, extrinsics and reprojection error ship alongside the episode.

• The streams on the rig match the streams in the spec, and no stream is assumed.
• Every stream is hardware-triggered on one PTP clock with nanosecond timestamps.
• The sync ceiling is stated as a number, with measured residuals reported rather than a hardware claim.
• Calibration is solved at the start and end of every session, and the drift is recorded.
• Intrinsics, extrinsics and reprojection error ship with each episode.
• Hand pose is delivered at 21 joints per hand with a per-joint visibility flag, if the spec calls for it.

Consent is a stage of its own because it cannot be repaired after capture. Every contributor signs a release before capture, and the same standard covers anyone who appears in frame incidentally, such as family members, coworkers and bystanders, not only the person performing the task under contract. Agree the consent and licensing basis during specification rather than after delivery.

Anonymization is part of the same stage. Faces, license plates and screens are detected, tracked and irreversibly blurred, with spoken identifiers scrubbed from audio, and a human reviewer confirms the result. The de-identification pass runs on every episode at ingest. There is no separate un-redacted master, and no step for you to run yourself. A batch data card records the detector version, pass timestamp and reviewer confirmation per episode, which is what makes the privacy claim checkable rather than asserted.

• Every contributor signs a release before capture.
• Anyone who appears in frame incidentally is covered by the same consent standard.
• Faces, plates and screens are detected, tracked and irreversibly blurred, and spoken identifiers are scrubbed from audio.
• A human reviewer confirms the de-identification pass on every episode.
• No un-redacted master is kept or delivered.
• The batch data card records detector version, pass timestamp and reviewer confirmation per episode.
• The consent chain is documented and warranted by the provider.

Ask a collection partner: Can I see the consent chain for a specific clip? Who reviews the blurring, and where is that review recorded? Is an un-redacted copy retained anywhere? A precise, written answer to each of these is what turns the consent claim into something you can check.

During capture, the question is whether what is being recorded is still what the spec asked for. Paid, consented contributors film first-person footage of real tasks, and recruitment draws on a global contributor pool screened against the cells named in the spec, so the population in the spec is the population captured. The capture tier is chosen to match the spec, not the other way around.

Three items are worth checking while capture runs. First, coverage: fill is measured against the condition cells at every batch, so a thin cell is visible early rather than at training time. Second, failure cases: if your policy needs to see stalls, slips and corrections, they have to be in the spec, because a dataset of only clean successes does not show a model what a recoverable mistake looks like. Third, calibration at both ends of each session, so that drift is detectable during the program rather than after it.

• Contributors are paid, consented and screened against the cells in the spec.
• The capture tier matches what the spec needs: phone, head-mounted rig or multi-camera rig.
• Coverage is reported against the condition budget at every batch.
• Failure and near-miss cases are captured wherever the spec requires them.
• Calibration is solved at the start and end of every session.
• Episodes beyond the sync ceiling are discarded at capture, not carried forward.

QA is where the spec becomes enforceable. Every Firsthand collection is validated against a written acceptance test from the skill spec. At ingest, episodes are machine-checked against reject criteria. A human reviewer then checks each episode against the task spec, and labels get a second-reviewer pass. Anything that misses is rejected, recorded in a reject log that ships with every batch, and recaptured at Firsthand’s cost rather than delivered with a caveat.

The unit that comes out of QA is the validated hour: 60 minutes of footage that has passed QA against your spec, with sync within the 2 ms ceiling, calibration solved and within drift tolerance, coverage cells filled and consent documented. It is not 60 minutes of raw recording. Only footage that clears the accept criteria is billed as a validated hour, which is why the reject log is the most useful single document to ask for: it shows how much footage was recorded against how much passed.

• The acceptance test is written into the spec and applied to every batch.
• Machine reject criteria run at ingest, and a human reviewer checks each episode against the task spec.
• Labels get a second-reviewer pass.
• A reject log with reasons ships with every batch.
• Rejected episodes are recaptured at the provider’s cost, not delivered with a caveat.
• Only footage that clears the accept criteria counts as a validated hour.

Delivery ships in the format agreed during specification, so the dataset loads into your pipeline without a bespoke parser or a reformatting pass. The formats stated on this site are RLDS, LeRobot, WebDataset, HDF5, zarr, Rerun, WAV and JSONL. For WebDataset, delivery is a set of .tar shards of 512 samples each, and each sample carries a key, head and wrist images, depth and an annotation file. For RLDS, shards are about 1.9 GB each. Every batch ships with the reject log and the batch data card.

Labels are delivered as the spec requires. The ones most orders ask for are hand_joints, a float32 array of shape [T, 2, 21, 3] in metres, so 21 joints per hand in 3D; hand_visible, the per-joint visibility flag; contact_events; and action_segments, a list of start_ns, end_ns, verb and noun entries that mark where each action begins and ends. Each is timestamped and aligned to every other stream.

• The batch arrives in the format and schema named in the spec.
• Every label named in the spec is present and aligned to the other streams.
• The reject log and batch data card are in the delivery.
• The license is perpetual, irrevocable, worldwide and buyer-owned, with consent chain documented and warranted.
• If the data must be collected only for you, exclusivity was requested at specification.

The table below condenses the questions from every stage into one list you can send to a provider before committing. A complete answer is specific, written and checkable. A vague answer is the signal to ask for the document behind it.

Most programs begin with a Spec Pilot: a small, fast first batch that proves the spec and the accept criteria hold up on real footage, with coverage, reject rate and a real timeline figure, before any commitment to scale. The flow is spec, Spec Pilot, then scale, and scaling runs the same pipeline with the same accept rules. A focused brief without a pilot is also a normal starting point, so a pilot is a decision to make, not a requirement.

The stated timeline begins after a signed spec. Discovery and specification come before it. Where contributor coverage is deep, such as home kitchens, retail handling and light assembly, first validated data arrives in a median of 48 hours from a signed spec. Where coverage is thin, such as construction, clinical, agriculture and hospitality, it is three to four weeks, because operators must be recruited and vetted. A time to full volume is not stated as a single figure, and your own internal approvals sit outside these numbers.

• You decided whether to run a Spec Pilot before committing to full volume.
• The pilot reports coverage, reject rate and a real timeline figure for your spec.
• You know whether your environments are deep-coverage or thin-coverage, and the timeline you are planning against.
• Your internal approvals are planned separately from the stated timeline.

A good checklist also records what the provider has not published, so that you ask for it instead of assuming it. The following items are not stated on this site. Raise each during the working call, once the spec is written down.