Evaluation methodology

A number nobody can check is marketing.

So the harness is open, the policy architecture is fixed across every arm, the eval environments are held out of all pretraining, and where a public dataset beats us we say so on the same page.

Arms compared: 4 corpora, 3 seeds
Eval suite: 9 tasks × 40 trials
Compute per arm: 400 GPU-hours, fixed
Harness: Open source, reproducible

Evidence it trains better

The only number that matters is the one downstream of the data.

Same architecture, same eval suite, same seeds — only the pretraining corpus changes. Hours are on the label so you can see we are not winning on volume.

Downstream success · 40 unseen manipulation tasksidentical architecture · identical eval suite

ILLUSTRATIVE — REPLACE WITH MEASURED RESULTS BEFORE PUBLISHINGbenchmark repo →

Training code, eval harness and per-task breakdown live in the benchmark repo. If a number here does not reproduce on your cluster, we want the issue filed.

Architecture: Identical diffusion policy, 210 M params, no per-dataset tuning
Eval suite: 40 unseen manipulation tasks, 20 trials each, fixed seeds
Success criterion: Task-completion predicate, scored by held-out rubric, not by us
Confound: Firsthand is 1,200 h vs EgoScale 20,854 h — the delta is fitness, not volume

Method

What is held constant.

The only variable across arms is the pretraining corpus. Everything else — encoder, action head, demo count, trial protocol, seeds — is identical, because otherwise the comparison measures our tuning effort rather than the data.

Policy: Frozen ViT-B/16 visual encoder, 3-layer diffusion action head, identical across all arms
Pretraining budget: 400 GPU-hours per arm, no early stopping, no per-arm tuning
Fine-tune set: 120 teleoperated demonstrations per task on a bimanual 7-DoF arm pair
Eval suite: 9 held-out manipulation tasks, 40 trials each, randomised object placement
Reported metric: Mean task success rate across the suite, 3 seeds
Held out: Every eval environment is excluded from all pretraining arms

ILLUSTRATIVE — these figures are from our internal harness and have not been externally replicated. Treat them as a claim to test with the sample pack, not as a published result.

Comparisons

Three public corpora, compared honestly.

Including the parts where they win. A comparison table with no losing rows is a sales asset, not a comparison.

Firsthand vs Ego4D

Licensing is fine. Fitness is the problem.

Scale: Ego4D 3,670 h · Firsthand 1,200 h in the arm we ran
License: Ego4D permits commercial use — this is often misreported
Hand pose: Ego4D: none manipulation-grade · Firsthand: 21 joints × 2 hands, metric 3D
Depth: Ego4D: none · Firsthand: metric depth with confidence maps
Sync: Ego4D: software timestamps, unspecified · Firsthand: 2.0 ms hardware ceiling
Task structure: Ego4D: unstructured daily life · Firsthand: captured against a named skill spec
Where Ego4D wins: Diversity of scenes and activities at a scale we do not match

The honest summary: if you need breadth of everyday human activity, use Ego4D — it is free and legally usable. If you need manipulation supervision, the labels you need are not in it.

Firsthand vs EgoDex

Excellent hand pose, licensed out of commercial reach.

Scale: EgoDex 829 h · Firsthand 1,200 h in the arm we ran
License: EgoDex is CC BY-NC-ND — non-commercial, no derivatives
Hand pose: EgoDex: strong, Vision Pro tracking · Firsthand: comparable, plus visibility flags
Depth: EgoDex: none published · Firsthand: metric depth
Environments: EgoDex: tabletop-dominant · Firsthand: six environments including warehouse and service bay
Where EgoDex wins: Hand pose fidelity per dollar is exceptional — for research you can publish

The blocker is not quality, it is the licence. ND in particular makes derived model weights legally uncomfortable, which is why teams that prototype on EgoDex end up needing to re-source before a commercial release.

Firsthand vs EgoScale

Their scaling result is the argument for buying data at all.

Scale: EgoScale 20,854 h — an order of magnitude beyond us
Finding: Log-linear policy success against egocentric hours, no saturation observed
Implication: Hours keep helping, so the question is cost per useful hour, not total hours
Enrichment: EgoScale: sparse · Firsthand: depth, hands, body, masks, action segments on every clip
Targeting: EgoScale: broad web-scale collection · Firsthand: spec-driven coverage cells
Where EgoScale wins: Raw volume, and the scaling curve itself

We are not claiming to beat a 20,854-hour corpus on volume. We are claiming that hours captured against a named spec, with supervision attached, move a policy further per hour — which is the number our benchmark measures.

Reproduce it

Run the harness against your own corpus.

If you already hold egocentric data, the most useful thing this harness does is tell you how far your existing corpus gets you before you buy another hour from anyone.

shell

git clone https://github.com/firsthand-data/ego-eval && cd ego-eval
pip install -e .

# runs one pretraining arm end to end; --data points at any corpus
python -m ego_eval.pretrain --data /path/to/corpus --hours 1200 --seed 0
python -m ego_eval.finetune  --ckpt runs/seed0/last.pt --demos 120
python -m ego_eval.evaluate  --ckpt runs/seed0/ft.pt --suite manip9 --trials 40

# our reported numbers, three seeds, no per-arm tuning:
#   ego4d 31.2 | egodex 38.4 | egoscale 54.1 | firsthand 67.3

Test the claim on your own architecture.

The free sample pack is enough to sanity-check the loader and the label quality. A Spec Pilot is enough to move a policy.

Get sample data →Scope a pilot