Error anticipation on RewardBench 2

The reading

1. internal features beat the exposed seen-pair margin at predicting held-out RB2 completion errors on Qwen3-8B

   err_auroc_delta_qwen8b > 0.03 0.6 -9.738e-4 Error anticipation on RewardBench 2 refuted registered confidence interval -0.004634508227426182 to 0.0023281954874443313 > 0.03 The frozen study "Error anticipation on RewardBench 2", adjudicated on 19 July 2026 at the registered level of the project's evidence ladder. One frozen study, adjudicated once, on skywork-v2-qwen3-8b and skywork-v2-llama31-8b, over rb2-full. It does not establish that the same holds on another subject, another dataset or another run. delta-AUROC point call at its own 0.03 threshold under the label-shuffle refit null (null sd 0.0235, null pass rate at threshold 0.2013); the registered CI exclusion carries size control and deltas at or above the MDE confirm at 80 percent. Open the record › Copy technical provenance refuted

2. the same holds on Llama-8B (cross-family)

   err_auroc_delta_llama8b > 0.03 -0.003471 Error anticipation on RewardBench 2 refuted registered confidence interval -0.005095063654071338 to -0.0018241777525145248 > 0.03 The frozen study "Error anticipation on RewardBench 2", adjudicated on 19 July 2026 at the registered level of the project's evidence ladder. One frozen study, adjudicated once, on skywork-v2-qwen3-8b and skywork-v2-llama31-8b, over rb2-full. It does not establish that the same holds on another subject, another dataset or another run. delta-AUROC point call at its own 0.03 threshold under the label-shuffle refit null (null sd 0.0235, null pass rate at threshold 0.2013); the registered CI exclusion carries size control and deltas at or above the MDE confirm at 80 percent. Open the record › Copy technical provenance refuted

internals add nothing over the seen-pair margin the RM already exposes when predicting held-out errors, on at least one family

err_auroc_delta_min_family <= 0 -0.003471 Error anticipation on RewardBench 2 refuted registered <= 0 The frozen study "Error anticipation on RewardBench 2", adjudicated on 19 July 2026 at the registered level of the project's evidence ladder. One frozen study, adjudicated once, on skywork-v2-qwen3-8b and skywork-v2-llama31-8b, over rb2-full. It does not establish that the same holds on another subject, another dataset or another run. delta-AUROC point call at its own 0.03 threshold under the label-shuffle refit null (null sd 0.0235, null pass rate at threshold 0.2013); the registered CI exclusion carries size control and deltas at or above the MDE confirm at 80 percent. Open the record › Copy technical provenance

Measurements

err_auroc_internal_qwen8b

0.8581 Error anticipation on RewardBench 2 refuted registered none registered The frozen study "Error anticipation on RewardBench 2", adjudicated on 19 July 2026 at the registered level of the project's evidence ladder. One frozen study, adjudicated once, on skywork-v2-qwen3-8b and skywork-v2-llama31-8b, over rb2-full. It does not establish that the same holds on another subject, another dataset or another run. Nothing was registered against this value in advance, so it is a supporting measurement rather than a tested one. delta-AUROC point call at its own 0.03 threshold under the label-shuffle refit null (null sd 0.0235, null pass rate at threshold 0.2013); the registered CI exclusion carries size control and deltas at or above the MDE confirm at 80 percent. Open the record › Copy technical provenance

err_auroc_margin_qwen8b

0.859 Error anticipation on RewardBench 2 refuted registered none registered The frozen study "Error anticipation on RewardBench 2", adjudicated on 19 July 2026 at the registered level of the project's evidence ladder. One frozen study, adjudicated once, on skywork-v2-qwen3-8b and skywork-v2-llama31-8b, over rb2-full. It does not establish that the same holds on another subject, another dataset or another run. Nothing was registered against this value in advance, so it is a supporting measurement rather than a tested one. delta-AUROC point call at its own 0.03 threshold under the label-shuffle refit null (null sd 0.0235, null pass rate at threshold 0.2013); the registered CI exclusion carries size control and deltas at or above the MDE confirm at 80 percent. Open the record › Copy technical provenance

err_auroc_delta_qwen8b_ci_low

-0.004635 Error anticipation on RewardBench 2 refuted registered confidence interval -0.004634508227426182 to 0.0023281954874443313 > 0.03 The frozen study "Error anticipation on RewardBench 2", adjudicated on 19 July 2026 at the registered level of the project's evidence ladder. One frozen study, adjudicated once, on skywork-v2-qwen3-8b and skywork-v2-llama31-8b, over rb2-full. It does not establish that the same holds on another subject, another dataset or another run. delta-AUROC point call at its own 0.03 threshold under the label-shuffle refit null (null sd 0.0235, null pass rate at threshold 0.2013); the registered CI exclusion carries size control and deltas at or above the MDE confirm at 80 percent. Open the record › Copy technical provenance

err_auroc_delta_qwen8b_ci_high

0.002328 Error anticipation on RewardBench 2 refuted registered confidence interval -0.004634508227426182 to 0.0023281954874443313 > 0.03 The frozen study "Error anticipation on RewardBench 2", adjudicated on 19 July 2026 at the registered level of the project's evidence ladder. One frozen study, adjudicated once, on skywork-v2-qwen3-8b and skywork-v2-llama31-8b, over rb2-full. It does not establish that the same holds on another subject, another dataset or another run. delta-AUROC point call at its own 0.03 threshold under the label-shuffle refit null (null sd 0.0235, null pass rate at threshold 0.2013); the registered CI exclusion carries size control and deltas at or above the MDE confirm at 80 percent. Open the record › Copy technical provenance

err_auroc_internal_llama8b

0.8728 Error anticipation on RewardBench 2 refuted registered none registered The frozen study "Error anticipation on RewardBench 2", adjudicated on 19 July 2026 at the registered level of the project's evidence ladder. One frozen study, adjudicated once, on skywork-v2-qwen3-8b and skywork-v2-llama31-8b, over rb2-full. It does not establish that the same holds on another subject, another dataset or another run. Nothing was registered against this value in advance, so it is a supporting measurement rather than a tested one. delta-AUROC point call at its own 0.03 threshold under the label-shuffle refit null (null sd 0.0235, null pass rate at threshold 0.2013); the registered CI exclusion carries size control and deltas at or above the MDE confirm at 80 percent. Open the record › Copy technical provenance

err_auroc_margin_llama8b

0.8763 Error anticipation on RewardBench 2 refuted registered none registered The frozen study "Error anticipation on RewardBench 2", adjudicated on 19 July 2026 at the registered level of the project's evidence ladder. One frozen study, adjudicated once, on skywork-v2-qwen3-8b and skywork-v2-llama31-8b, over rb2-full. It does not establish that the same holds on another subject, another dataset or another run. Nothing was registered against this value in advance, so it is a supporting measurement rather than a tested one. delta-AUROC point call at its own 0.03 threshold under the label-shuffle refit null (null sd 0.0235, null pass rate at threshold 0.2013); the registered CI exclusion carries size control and deltas at or above the MDE confirm at 80 percent. Open the record › Copy technical provenance

err_auroc_delta_llama8b_ci_low

-0.005095 Error anticipation on RewardBench 2 refuted registered confidence interval -0.005095063654071338 to -0.0018241777525145248 > 0.03 The frozen study "Error anticipation on RewardBench 2", adjudicated on 19 July 2026 at the registered level of the project's evidence ladder. One frozen study, adjudicated once, on skywork-v2-qwen3-8b and skywork-v2-llama31-8b, over rb2-full. It does not establish that the same holds on another subject, another dataset or another run. delta-AUROC point call at its own 0.03 threshold under the label-shuffle refit null (null sd 0.0235, null pass rate at threshold 0.2013); the registered CI exclusion carries size control and deltas at or above the MDE confirm at 80 percent. Open the record › Copy technical provenance

err_auroc_delta_llama8b_ci_high

-0.001824 Error anticipation on RewardBench 2 refuted registered confidence interval -0.005095063654071338 to -0.0018241777525145248 > 0.03 The frozen study "Error anticipation on RewardBench 2", adjudicated on 19 July 2026 at the registered level of the project's evidence ladder. One frozen study, adjudicated once, on skywork-v2-qwen3-8b and skywork-v2-llama31-8b, over rb2-full. It does not establish that the same holds on another subject, another dataset or another run. delta-AUROC point call at its own 0.03 threshold under the label-shuffle refit null (null sd 0.0235, null pass rate at threshold 0.2013); the registered CI exclusion carries size control and deltas at or above the MDE confirm at 80 percent. Open the record › Copy technical provenance

err_auroc_delta_min_family

-0.003471 Error anticipation on RewardBench 2 refuted registered <= 0 The frozen study "Error anticipation on RewardBench 2", adjudicated on 19 July 2026 at the registered level of the project's evidence ladder. One frozen study, adjudicated once, on skywork-v2-qwen3-8b and skywork-v2-llama31-8b, over rb2-full. It does not establish that the same holds on another subject, another dataset or another run. delta-AUROC point call at its own 0.03 threshold under the label-shuffle refit null (null sd 0.0235, null pass rate at threshold 0.2013); the registered CI exclusion carries size control and deltas at or above the MDE confirm at 80 percent. Open the record › Copy technical provenance

Provenance

card

ERR-RB2

frozen_at

2026-07-18

adjudicated_at

2026-07-19

spec_hash

spec:c8ac36e72adcc96a3d65b802efbce423

study_id

study:campaign-err-rb2@v1#c8ac36e7

Evidence records

6bf316c059230ff4ff99b118b1664631

power_acceptance

delta-AUROC point call at its own 0.03 threshold under the label-shuffle refit null (null sd 0.0235, null pass rate at threshold 0.2013); the registered CI exclusion carries size control and deltas at or above the MDE confirm at 80 percent

spec_file

campaign-err-rb2.json