How we decide a result is real.
A clear evidence standard sits between the lab and every claim. Each result carries how it was produced — rigor is built into the pipeline, not added after.
The gates
Evidence ladder
Every result is tagged with how it was produced, so we always know what a number is allowed to mean.
Data contracts
Before the first hardware run, we fixed the data contracts every real dataset must follow — row schema, feature shapes, run-bundle manifests, scenario-held-out splits, and the adversarial checks that gate every result. That is the standard the bench now records against.
Schema: fixed sample period · scenario-held-out splits · manifest + samples per run bundle · filenames carry no labels.
The adversarial checks are not optional extras: a label-shuffle model must collapse to chance and a metadata-only model must fail, or the result is rejected before it can claim anything.
Controlled run bundles are being recorded on the bench Bench. Dataset cards will be published here only after labeled runs pass their truth gates — recordings come first; the publishable result waits on the gates.
The AER-TRUTH protocol
The evidence standard on this page is the AER-TRUTH protocol: it tags every result with the stage it was produced at, so a number can only ever claim what its stage supports. The spec is in preparation; the standard is already running in the lab.
Where we are now
Every result on this site carries one of these tags. If a result isn't tagged, we haven't earned it yet.
Further reading
- →Machine smell — the array the recorded runs come from.
- →Data contracts — the schema and adversarial checks the gates enforce.
- →Our research pillars