Describes library 3.0.0, the current release.

subject: Skywork/Skywork-Reward-Llama-3.1-8B-v0.2
reproduction: No independent reproduction recorded.

Where does a preference live inside a reward model?

This model prefers one answer over another. Where does that preference live, and if I intervene there, does the score move?

subject: Skywork/Skywork-Reward-Llama-3.1-8B-v0.2
access: forward and backward, to weights
trust: exploratory

What was measured, and with what access

Skywork-Reward-Llama-3.1-8B-v0.2, twelve objective dimensions, forward and backward access to the model. Components ranked twice: once by observational attribution, once by causal patching.

What was measured, and with what access
objective dimensions	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl
preference pairs per dimension	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl
components attributed	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl
components patched	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl

Rank it two ways and watch them disagree

confidence

Pick a component and patch it. The original score stays on the axis.

attribution rank

patch rank

reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl
reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl
reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl
reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl
reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl
reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl
reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl
reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl
reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl
reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl

reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl
reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl
reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl
reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl
reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl
reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl
reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl
reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl
reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl
reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl

ranked beyond this list mlp_L31 reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl mlp_L30 reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl mlp_L29 reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl mlp_L28 reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl mlp_L27 reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl mlp_L26 reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl attn_L31 reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl mlp_L23 reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl mlp_L22 reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl mlp_L25 reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl

attribution rank 1, patch rank 64, moves 0.0000%. Recorded at reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl. attribution rank 2, patch rank 61, moves 0.1254%. Recorded at reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl. attribution rank 3, patch rank 43, moves 0.9455%. Recorded at reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl. attribution rank 4, patch rank 37, moves 1.5236%. Recorded at reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl. attribution rank 5, patch rank 41, moves 1.0242%. Recorded at reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl. attribution rank 6, patch rank 40, moves 1.1239%. Recorded at reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl. attribution rank 7, patch rank 63, moves 0.0000%. Recorded at reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl. attribution rank 8, patch rank 48, moves 0.7424%. Recorded at reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl. attribution rank 9, patch rank 44, moves 0.8379%. Recorded at reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl. attribution rank 10, patch rank 39, moves 1.1590%. Recorded at reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl. attribution rank 62, patch rank 1, moves 56.7073%. Recorded at reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl. attribution rank 39, patch rank 2, moves 16.7828%. Recorded at reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl. attribution rank 48, patch rank 3, moves 15.8350%. Recorded at reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl. attribution rank 64, patch rank 4, moves 12.7580%. Recorded at reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl. attribution rank 19, patch rank 5, moves 12.8007%. Recorded at reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl. attribution rank 63, patch rank 6, moves 12.9729%. Recorded at reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl. attribution rank 32, patch rank 7, moves 11.1624%. Recorded at reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl. attribution rank 17, patch rank 8, moves 9.2923%. Recorded at reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl. attribution rank 43, patch rank 9, moves 10.7626%. Recorded at reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl. attribution rank 34, patch rank 10, moves 10.7171%. Recorded at reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl.

margin

margin mean reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl

components

The two rankings point in different directions

Where a preference appears is not what causally changes it.

Observational localization and causal localization can point in different directions on the same model and the same objective. A component that a visible representation makes prominent is a hypothesis about mechanism. It is not a causal result until something is patched.

Spearman rank correlation reward-lens-assay/fixtures/e_parity/golden.json:9 across twelve dimensions on this model. Eight of the twelve dimensions are negative and four are positive.

On a second reward model the same measurement is near zero.

The same measurement on a second reward model gives a relationship near zero rather than a negative one. That is the reason this is a statement about the two methods disagreeing and not a statement that attribution is wrong: on one model the rankings oppose each other, on another they are close to unrelated, and neither pattern licenses preferring one localizer.

Attribution rank against patch-effect rank
mean rank correlation, Skywork/Skywork-Reward-Llama-3.1-8B-v0.2	reward-lens-assay/fixtures/e_parity/golden.json:9
mean rank correlation, ArmoRM-Llama3-8B-v0.1	reward-lens-assay/fixtures/e_parity/golden.json:6

Fig. 1.2 · Attribution rank against patch-effect rank One model, twelve dimensions, one frozen study.

reward-lens-assay/fixtures/e_parity/golden.json:54

positive dimensions · negative dimensions
dimension	spearman rho
code_correctness	reward-lens-assay/fixtures/e_parity/golden.json:54
confidence	reward-lens-assay/fixtures/e_parity/golden.json:54
correctness	reward-lens-assay/fixtures/e_parity/golden.json:54
factuality	reward-lens-assay/fixtures/e_parity/golden.json:54
formatting	reward-lens-assay/fixtures/e_parity/golden.json:54
helpfulness	reward-lens-assay/fixtures/e_parity/golden.json:54
instruction_following	reward-lens-assay/fixtures/e_parity/golden.json:54
math_correctness	reward-lens-assay/fixtures/e_parity/golden.json:54
refusal_quality	reward-lens-assay/fixtures/e_parity/golden.json:54
safety	reward-lens-assay/fixtures/e_parity/golden.json:54
sycophancy	reward-lens-assay/fixtures/e_parity/golden.json:54
verbosity	reward-lens-assay/fixtures/e_parity/golden.json:54

Twelve objective dimensions on one model. Above the zero line the two methods agree on a component's importance; below it they disagree. Eight of twelve fall below.

What this does not establish

This does not establish that attribution is wrong and patching is right. On a model organism with a planted answer key, the prediction that patching would recover the key better than attribution was frozen before the data, and the result contradicted it. The recorded conclusion is that neither localization can be preferred. A noisy answer key is the disclosed alternative reading and it has not been ruled out.

The two rankings do not match.

Attribution rank against patch-effect rank
	measured value	patch rank
code_correctness, mlp_L31	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl
confidence, mlp_L31	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl
correctness, mlp_L31	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl
factuality, mlp_L31	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl
formatting, mlp_L31	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl
helpfulness, mlp_L31	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl
instruction_following, mlp_L31	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl
math_correctness, mlp_L31	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl
refusal_quality, mlp_L31	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl
safety, mlp_L31	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl
sycophancy, mlp_L31	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl
verbosity, mlp_L31	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl

Attribution rank against patch-effect rank

Attribution rank against patch-effect rank
	measured value	patch rank
code_correctness	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl
confidence	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl
correctness	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl
factuality	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl
formatting	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl
helpfulness	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl
instruction_following	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl
math_correctness	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl
refusal_quality	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl
safety	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl
sycophancy	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl
verbosity	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl

One model, twelve dimensions, one frozen study.

The two rankings are not taken over the same set of components. Attribution covers one more component than patching does, because the embedding was attributed and never patched, so the correlation is computed on the smaller set and the embedding appears in neither ranking below.

the published correlation, and the correction to it

The project published one correlation and then corrected it against its own fixtures. Both values are parsed out of the correction note rather than transcribed, because the correction is the content.

the published correlation, and the correction to it
published, then deprecated	reward-lens-clean/docs/BUILD_NOTES.md:44
superseding value, as quoted	reward-lens-clean/docs/BUILD_NOTES.md:50
recomputed from the fixture	reward-lens-assay/fixtures/e_parity/golden.json:9
stated tolerance	reward-lens-assay/fixtures/e_parity/golden.json:9

the planted-key calibration case

One calibration case exists where the answer is known by construction: a model organism with a planted key, and a prediction frozen before the data that causal patching would recover that key better than attribution. It did not. Both recovery scores are below, against chance, with the project's own recorded reading of them.

The claim, as it was frozen patching's ranking matches the organism's marker-responsive components (the twin-activation-delta answer key) better than attribution's

the planted-key calibration case
method	key recovery, area under the curve
attribution	planted-key calibration case, attribution against patching refuted frozen before the data attribution_recovery_auc 1 patching_recovery_auc 0.4339622641509434 real8b_avp_spearman -0.3735805860805861 recovery_gap -0.5660377358490566 recovery_gap_ci_high -0.4276729559748428 recovery_gap_ci_low -0.7735849056603774 f93f4b5578c221d3d2665f9b305f3e86759be862+dirty Open the record ›
causal patching	planted-key calibration case, attribution against patching refuted frozen before the data attribution_recovery_auc 1 patching_recovery_auc 0.4339622641509434 real8b_avp_spearman -0.3735805860805861 recovery_gap -0.5660377358490566 recovery_gap_ci_high -0.4276729559748428 recovery_gap_ci_low -0.7735849056603774 f93f4b5578c221d3d2665f9b305f3e86759be862+dirty Open the record ›
reference: chance	0.5

Treat attribution as a hypothesis generator, never as evidence of cause. If you have not patched, you do not have a causal result, no matter how clean the attribution bar looks.

Run it yourself

bash

unzip reward-model.zip -d reward-model && python reward-model/reproduce.py

Reproduce this

Everything on this page regenerates from the bundle below.

contents: attribution_per_pair.jsonl, golden.json, patching_per_pair.jsonl, reproduce.py
size: public/bundles/reward-model.zip
sha256: ed9073f8c4343668fa4091a25fe62c6f5b18a642f78629ec3b087787b0bd6661
commit: public/bundles/reward-model.zip
needs: Python 3.10 or newer. No GPU, no model download and no network. This reproduces the rank correlation from the emitted fixture. Reproducing the patching run itself needs weight access and a GPU, and the bundle does not contain the model.

Download the data

Expected output

pairs joined                360
components ranked both ways 64
dimensions                  12

dimension              recomputed  published   top attribution  moves
code_correctness       -0.441117  -0.441117   mlp_L31         7.0788%
confidence             -0.381206  -0.381206   mlp_L31         0.0000%
correctness            +0.117822  +0.117822   mlp_L31        34.2073%
factuality             -0.258911  -0.258911   mlp_L31         2.3732%
formatting             +0.076205  +0.076205   mlp_L31       235.2342%
helpfulness            -0.305981  -0.305981   mlp_L31         5.0162%
instruction_following  +0.015099  +0.015099   mlp_L31        18.1439%
math_correctness       +0.152893  +0.152893   mlp_L31        43.1881%
refusal_quality        -0.353836  -0.353836   mlp_L31         0.0000%
safety                 -0.289653  -0.289653   mlp_L31         4.0281%
sycophancy             -0.334428  -0.334428   mlp_L31         0.0000%
verbosity              -0.047506  -0.047506   mlp_L31        18.1041%

largest disagreement with the fixture  0.000000  (tolerance 0.01)
within tolerance                       yes

No independent reproduction recorded.

	measured value	of
models	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl	not measured
objective dimensions	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl	not measured
preference pairs per dimension	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl	not measured
attribution rows read	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl	not measured
patching rows read	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl	not measured
patching mode	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl	not measured
components attributed but not patched	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/patching_per_pair.jsonl	not measured
code_correctness, pairs with a negative margin	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl
confidence, pairs with a negative margin	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl
correctness, pairs with a negative margin	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl
factuality, pairs with a negative margin	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl
formatting, pairs with a negative margin	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl
helpfulness, pairs with a negative margin	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl
instruction_following, pairs with a negative margin	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl
math_correctness, pairs with a negative margin	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl
refusal_quality, pairs with a negative margin	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl
safety, pairs with a negative margin	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl
sycophancy, pairs with a negative margin	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl
verbosity, pairs with a negative margin	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl	reward-lens/outputs/v2_20260506_222648_unknown/e04_faithfulness_population/Skywork-Reward-Llama-3.1-8B-v0.2/attribution_per_pair.jsonl