What this does not do
Stated once, so nobody has to discover it in review.
This project is early. One person built it, and no result on this site has been independently reproduced. Everything below is what that means concretely.
What is not being claimed
-
Not the Goodhart turning point.
That is occupied, with a solver. What is measured here is the turn in a named gold signal, and if that signal is itself a proxy for something inexpressible, which it always is, then a measured turn is the turn for that signal.
-
Not white-box superiority.
The bar is decorrelation plus signal, reported against a scaffolded black-box baseline every time. Three published head-to-heads put white-box last, and the correlation between a white-box instrument's errors and a black-box baseline's errors is a required output field.
-
Not that nobody predicts reward hacking.
At least four methods do, one of which detects a latent propensity persisting while the behaviour is suppressed.
-
Not that intervals are absent from the field.
A government evaluator published 95% confidence intervals six days before this project's own field scan ran.
-
Not that monitors degrading under pressure is a new observation.
It is a small live subfield with results pointing in both directions. What is missing is turning that degradation curve into the figure of merit by which competing monitors are ranked.
-
Not linear-representation realism.
The portfolio is deliberately built so that most of it survives that hypothesis failing completely.
What is not built
| What it measures | implemented | open |
|---|---|---|
| Signal validity: the grader as a measurement device | src/data/atlas.json | src/data/atlas.json |
| Grader structure: is this thing a scalar, and what is it made of? | src/data/atlas.json | src/data/atlas.json |
| Grader white-box: the existing battery, re-typed | src/data/atlas.json | src/data/atlas.json |
| Verifier and environment science | src/data/atlas.json | src/data/atlas.json |
| The estimator: where a good reward becomes a bad gradient | src/data/atlas.json | src/data/atlas.json |
| The four books | src/data/atlas.json | src/data/atlas.json |
| Credit geometry | src/data/atlas.json | src/data/atlas.json |
| Rate and regime | src/data/atlas.json | src/data/atlas.json |
| Adversarial pressure, and the monitor as a target | src/data/atlas.json | src/data/atlas.json |
| Monitoring | src/data/atlas.json | src/data/atlas.json |
| Transfer and survival | src/data/atlas.json | src/data/atlas.json |
| Reference materials and label validity | src/data/atlas.json | src/data/atlas.json |
| Meta-instruments: measurements about measurements | src/data/atlas.json | src/data/atlas.json |
| The frontier: what would happen if we optimised | src/data/atlas.json | src/data/atlas.json |
src/data/atlas.json of the registry's instruments are implemented and src/data/atlas.json are specified and not built.
Specified and not built. Listed here at the same weight as the rest, because a catalogue that quietly separates what exists from what is planned is one a reader stops trusting the moment they notice.
Implemented means the code exists. It does not mean it has been run, and it does not mean it has been run on anything real. The build record keeps those apart across src/data/atlas.json capabilities, and src/data/atlas.json of them were exercised on a GPU rather than on a stub or a CPU fixture.
| how many | |
|---|---|
| PASS | src/data/atlas.json |
| NOT-RUN | src/data/atlas.json |
| EXCUSED | src/data/atlas.json |
| PASS-DEGRADED | src/data/atlas.json |
| FAIL | src/data/atlas.json |
A reading is a different thing again. The archive keeps src/data/atlas.json records and src/data/atlas.json of them are calibrated against a reference material, so no instrument on this site has been qualified in the sense the library defines. Every archived record sits at the lowest evidence strength.
What was claimed, and how it came out, is a fourth thing. The archive froze its predictions before the runs and kept every outcome, including the ones that went nowhere.
| how many | |
|---|---|
| inconclusive | src/data/atlas.json |
| refuted | src/data/atlas.json |
| mixed | src/data/atlas.json |
| confirmed | src/data/atlas.json |
No independent reproduction has been recorded.
The strongest experiment this project could run does not exist yet.