reward-lens

Describes library 3.0.0, the current release.

subject
ai-safety-institute/reward-hacking-olmo3.1-32b-kl0.0-seed2
reproduction
No independent reproduction recorded.

Is the objective drifting while the run is still going?

My policy is training right now. Is the objective it is actually optimizing drifting away from the one I wrote, and will I know before it is obvious?

rollouts
reward-lens-assay/PREDICTIONS.md:154
steps, lower of two records
reward-lens-assay/PREDICTIONS.md:154
rollouts per step
reward-lens-assay/PREDICTIONS.md:154
reproduction
No independent reproduction recorded.

What was measured, and with what access

A public labelled reinforcement-learning run with reward hacking: reward-lens-assay/PREDICTIONS.md:154 labelled rollouts across reward-lens-assay/PREDICTIONS.md:154 training steps, or reward-lens-assay/PREDICTIONS.md:154 by the run's other record, which is the count its own rollout table carries. The hacking transition is fitted at step reward-lens-assay/PREDICTIONS.md:154 with a 10-to-90 width of reward-lens-assay/PREDICTIONS.md:154 steps.

What was measured, and with what access
labelled rollouts reward-lens-assay/PREDICTIONS.md:154
training steps, the lower of two records reward-lens-assay/PREDICTIONS.md:154
fitted transition midpoint reward-lens-assay/PREDICTIONS.md:154
10-90 transition width reward-lens-assay/PREDICTIONS.md:154
fit quality, R squared reward-lens-assay/PREDICTIONS.md:154

Move through the run and watch the signals

Read each row across before choosing one. The signal with the longest apparent lead is also the one that fires most often on order-destroyed surrogates of the same series, which is why it was not the one kept. Rows stay in the study's own evaluation order and are never sorted by lead.

Training step. Arrow keys step by one, page keys by ten, home and end go to the ends.
detector lead, transition widths fires on surrogates reward-lens-assay/PREDICTIONS.md:154 reward-lens-assay/PREDICTIONS.md:154 reward-lens-assay/PREDICTIONS.md:154 reward-lens-assay/PREDICTIONS.md:154 reward-lens-assay/PREDICTIONS.md:154 reward-lens-assay/PREDICTIONS.md:154
step
reward-lens-assay/experiments/x5_threshold/aisi_lengths.npz
hack rate
reward-lens-assay/experiments/x5_threshold/aisi_lengths.npz
alarm
reward-lens-assay/PREDICTIONS.md:154
midpoint
reward-lens-assay/PREDICTIONS.md:154
comparator
reward-lens-assay/PREDICTIONS.md:154

Two records of this run report its length differently, as reward-lens-assay/PREDICTIONS.md:154 steps and as reward-lens-assay/PREDICTIONS.md:154. This site uses reward-lens-assay/PREDICTIONS.md:154 and says so here rather than choosing one quietly. The public rollout table labels its steps from one, which accounts for the difference arithmetically without establishing which record is right.

One signal led. The comparator named in advance lagged.

The shipped signal leads. The comparator named in advance lags.

The variance derivative alarms at step reward-lens-assay/PREDICTIONS.md:154, a lead of reward-lens-assay/PREDICTIONS.md:154 transition widths. The gradient-norm comparator, named in advance, peaks at step reward-lens-assay/PREDICTIONS.md:154, which is reward-lens-assay/PREDICTIONS.md:154 steps after the midpoint, a lead of reward-lens-assay/PREDICTIONS.md:154 widths.

Two signals that lead by more, and why they were discarded.

The variance level alarms at step reward-lens-assay/PREDICTIONS.md:154 for a lead of reward-lens-assay/PREDICTIONS.md:154 widths and fires on reward-lens-assay/PREDICTIONS.md:154 of order-destroyed surrogates from the same series. The gradient norm under a cumulative-sum detector fires on reward-lens-assay/PREDICTIONS.md:154. The shipped derivative fires on reward-lens-assay/PREDICTIONS.md:154.

Fig. 2.3 · The same detectors, on the real run and on order-destroyed surrogates One run. The false-alarm rate across runs is not settled.
fires on surrogates lead, transition widths not measured not measured Within-group reward variance, level Gradient norm under a cumulative-sum alarm Within-group reward variance, derivative Gradient-norm peak, the comparator named in advance
lead, transition widths · fires on surrogates
detector lead, transition widths fires on surrogates
Within-group reward variance, level reward-lens-assay/PREDICTIONS.md:154 reward-lens-assay/PREDICTIONS.md:154
Gradient norm under a cumulative-sum alarm reward-lens-assay/PREDICTIONS.md:154
Within-group reward variance, derivative shipped reward-lens-assay/PREDICTIONS.md:154 reward-lens-assay/PREDICTIONS.md:154
Gradient-norm peak, the comparator named in advance reward-lens-assay/PREDICTIONS.md:154

A surrogate keeps every value of the real series and destroys the order. A detector that fires as often on the surrogate as on the real run is reading the distribution, not the drift.

What this does not establish

This is one scalar series on one public run. No activations were read and no mechanism is named. The false-positive rate across runs is not settled, and the gradient-norm comparator could not be computed on the companion analysis at all, because the published rollout table carries no optimiser telemetry.

What this does not establish

The same detectors, on the real run and on order-destroyed surrogates
fires on surrogates, Within-group reward variance, derivative reward-lens-assay/PREDICTIONS.md:154
fires on surrogates, from the trainer log reward-lens-assay/PREDICTIONS.md:154
fires on surrogates, Within-group reward variance, level reward-lens-assay/PREDICTIONS.md:154
fires on surrogates, Gradient norm under a cumulative-sum alarm reward-lens-assay/PREDICTIONS.md:154
fires on surrogates, Gradient-norm peak, the comparator named in advance

A defect this build found in its own frozen analysis.

The frozen path for a companion prediction standardised the series against a mean the post-transition regime had raised, so it used the future to define what normal looked like and reported a large lead that is an artifact. A detector that uses the future to define normal cannot measure a lead time, and it will report one. Both numbers are published, they disagree in sign, and the flattering one was declined.

The frozen path standardises the series against a mean the post-transition regime raised, so the accumulator crosses a threshold defined partly by the future. A detector that uses the future to define normal cannot measure a lead time, and it will report one. The detector-free comparison was added after the freeze, and it is labelled as such wherever it appears.

A defect this build found in its own frozen analysis.
the frozen path reward-lens-assay/PREDICTIONS.md:136
the corrected path reward-lens-assay/PREDICTIONS.md:136
flattering value declined reward-lens-assay/PREDICTIONS.md:136

The companion prediction on the same run, which went the other way.

A second quantity, named in the same freeze, was predicted to move before the labelled hack rate. It lags, by reward-lens-assay/PREDICTIONS.md:155 transition widths. The sign is robust and the magnitude is not. What failed is the lead-time claim and not the instrument.

The companion prediction on the same run, which went the other way.
Lambda moves before the labelled hack rate reward-lens-assay/PREDICTIONS.md:155
sign robust reward-lens-assay/PREDICTIONS.md:155
magnitude robust reward-lens-assay/PREDICTIONS.md:155
the frozen path
reward-lens-assay/PREDICTIONS.md:136
the corrected path
reward-lens-assay/PREDICTIONS.md:136

The frozen path standardises the series against a mean the post-transition regime raised, so the accumulator crosses a threshold defined partly by the future. A detector that uses the future to define normal cannot measure a lead time, and it will report one. The detector-free comparison was added after the freeze, and it is labelled as such wherever it appears.

Run it yourself

bash 
unzip run.zip -d run && python run/reproduce.py

Reproduce this

Everything on this page regenerates from the bundle below.

contents
PREDICTIONS-R3-R4.txt, aisi_lengths.npz, reproduce.py, run-series.json
size
public/bundles/run.zip
sha256
ba9b5eba3ac2161016b5bdcec24d2a39cd6a7dff496792805f3438f7dd3852ad
commit
public/bundles/run.zip
needs
Python 3.10 or newer. No GPU, no model and no network. The bundle carries the labelled rollout table, so the fit regenerates end to end from what is in it. The alarm does not: the rollout table records lengths and outcomes and carries no reward and no group index, so a within-group reward variance cannot be formed from it. The bundle re-fits the transition and prints its own numbers beside the ledger's.

Download the data

Expected output

rollouts in the table       25664
rollouts in run-series.json 25664
distinct step indices       401  (1 to 401)
rollouts per step           64
step count the site states  400

series formatOk    401 points   matches run-series.json: true
series hackRate    401 points   matches run-series.json: true
series realized    401 points   matches run-series.json: true
series written     401 points   matches run-series.json: true

                       this fit    frozen ledger
transition midpoint    106.9368    106.0000
10-to-90 width          23.9823     23.9000
R2                      0.99555     0.99600
floor and ceiling      0.0022 to 0.9892

detector alarms, from the frozen ledger row in this bundle, not refitted here
  variance_level         step 70     lead +1.504 widths
  gradient_norm_cusum    no alarm  
  variance_derivative    step 90     lead +0.668 widths  <- shipped
  gradient_norm_peak     step 133    lead -1.129 widths

No independent reproduction recorded.