Research · Method
The protocol is hashed before the run
The published figures come from one bench: the served engine, run on cases whose outcome is known, scored by a judge fixed in advance. Measured 2026-09-29 on the development lot, engine commit 7a86f874.
Pre-registration
A protocol file sealed by its hash
The protocol file states what the engine receives, how the lots are built, the score of each family and the pass thresholds. It is written before the first run on a lot and sealed by its SHA-256. The runner refuses to start when the file no longer matches its seal. A later change is a dated amendment, sealed again.
- Protocol hash
5e4f94787d87e7fdeea69b5cb0978a4af0859cdbcc3400e5b3d38ad246c642eb
Registry
Each full run writes one registry line
A registry line holds the timestamp, the lot, the engine commit and branch, any uncommitted change, and the hashes of the protocol, the lots and the judge code. Smoke runs stay out of it. Two engine commits are compared when the judge hash and the lots hash are equal.
- Registered runs
- 26 lines, control runs included
- Decision families
- 13 families in the bench
The engine receives what a client would have: the sample, the decision, the history before the origin date. The outcome stays with the judge, and a test checks that the engine input is unchanged when an outcome changes.
Controls
The bench is checked before the engine
- Oracle
- a control that knows the outcome scores exactly one in each family
- Coin flip
- a control that answers at random scores zero or less in each family
- Graded
- a control right on a fixed share of cases scores close to that share
- Placebo
- the served engine, given the input of another case of the same shape, falls well below the served score
- Null baselines
- each family has simple baselines, and the engine is scored against the best of them
Calibration
Stated probabilities checked against observed frequencies
The calibration error is computed in ten equal classes, pooled over the probabilities the engine served. A perfect forecaster scores above zero on a finite sample, so the error is read against a floor simulated under perfect calibration. The threshold applies to the excess over that floor.
- Error
- 0.0107 over 75,998 probabilities served
- Simulated floor
- 0.0018 for a perfect forecaster on the same cases
- Excess
- 0.0089, against a pass threshold of 0.03
Coverage
Realised values inside the announced intervals
For the families where the engine serves a forecast range, the bench counts the share of realised values inside the central interval. Coverage is read family by family.
| Family | Cases | 80% interval | 90% interval |
|---|---|---|---|
| Level forecasts | 300 | 80.5% | 90.0% |
| Cold-start forecasts | 600 | 79.1% | 89.5% |
| Monthly failure counts | 105 | 82.3% | 92.9% |
Measured 2026-09-29 on the development lot, engine commit 7a86f874. Source: the engine run registry, entry 26.
Silence
A named refusal counts as a silence
The engine can decline, and each refusal is tallied by its code. In the score, a silence leaves the client with the baseline. Speaking and being wrong costs as much as speaking and being right earns. A date guard stays on in each attempt. On this run, no winner was returned on 0.2% of cases.
Confirmation
The confirmation lot stays sealed until release
The development lot counts progress from one commit to the next. A confirmation lot accepts no partial run and asks for a declared reason before it opens; each opening enters the registry. The historical confirmation lot was read during development, which the protocol declares. The release decision rests on one declared reading of a fresh sealed lot.