reward-lens

Describes library 3.0.0, the current release.

subjects
four scoring programs
reproduction
No independent reproduction recorded.

Is the grader underneath all of this sound?

Everything above depends on a grader producing scores. Is the grader sound?

subjects
four scoring programs
access
full, log_only
trust
exploratory

What was measured, and with what access

Four scoring programs in wide use: a mathematical equivalence checker from a standard maths benchmark, the report builder inside a software-engineering benchmark's grading module, and two environment reward functions that ship with a widely used trainer.

What was measured, and with what access
subjects reward-lens--github/reward-lens-assay/experiments/x1_release/cards/is_equiv.txt
fields per card reward-lens--github/reward-lens-assay/experiments/x1_release/cards/is_equiv.txt
hendrycks/math is_equiv reward-lens--github/reward-lens-assay/experiments/x1_release/cards/is_equiv.txt reward-lens--github/reward-lens-assay/experiments/x1_release/cards/is_equiv.txt
SWE-bench grading.py reward-lens--github/reward-lens-assay/experiments/x1_release/cards/swebench.txt reward-lens--github/reward-lens-assay/experiments/x1_release/cards/swebench.txt
verl GSM8K reward reward-lens--github/reward-lens-assay/experiments/x1_release/cards/verl_gsm8k.txt reward-lens--github/reward-lens-assay/experiments/x1_release/cards/verl_gsm8k.txt
verl Search-R1 QA exact match reward-lens--github/reward-lens-assay/experiments/x1_release/cards/verl_search_r1.txt reward-lens--github/reward-lens-assay/experiments/x1_release/cards/verl_search_r1.txt

Read a card, and watch most of it decline

full

hendrycks/math is_equiv

exploratory

grader MATH-500 gold answers, 240 rollouts

https://raw.githubusercontent.com/hendrycks/math/main/modeling/math_equivalence.py (sha256 c4101b1f51a2bb65)

reward-lens--github/reward-lens-assay/experiments/x1_release/cards/is_equiv.txt read reward-lens--github/reward-lens-assay/experiments/x1_release/cards/is_equiv.txt refused The lowest trust among them is exploratory.

coverage
Across 240 rollouts is_equiv has never taken 32% of its branches (12 of 38). Those are behaviours it cannot distinguish.
  • never fired: _fix_fracs.clause4, _fix_fracs.clause5, _remove_right_units.clause1, _strip_string.clause1, _strip_string.clause5, is_equiv.clause1, is_equiv.clause2, is_equiv.clause3
  • a random sample of the same size reaches 0.653 against the corpus's 0.684
surviving mutants
Of 60 mutants of is_equiv, 60 survive: 60 ways it could be wrong that no rollout in your corpus would reveal.
  • mutation score 0.000 over 60 competent
  • reproducers withheld. this payload enumerates concrete ways to make the grader wrong. It is written to the store with a sensitive flag, excluded from rendered artifacts unless explicitly requested, and not published without a recorded decision.
metamorphic violations
metamorphic violations on grade
  • 259 violations over 378 applicable rollout-relation pairs (68.52%)
  • corpus 240 rollouts, 120 accepted at score > 0
  • baseline (identity transformation): 0 violations
  • (+4 more lines in this instrument's own report)
sensitivity profile
RECORD_INCOMPLETE declined

the record this card was built from carries no rubric scorer over named numeric inputs and no declared input ranges

expose the grader as a scorer over named numeric inputs and declare their ranges: `CardInputs(scorer=fn, rubric_inputs=[RubricInput('helpfulness', 0.0, 1.0), ...])`. A Sobol decomposition apportions output variance across an input space, so a grader with no numeric input space has no sensitivity profile rather than a flat one.

false-positive catalogue
RECORD_INCOMPLETE declined

the record this card was built from carries no stricter reference oracle and no seed rollouts to search around

supply a callable grader, a stricter reference that decides the same question, and seed rollouts to search around: `CardInputs(grader=fn, reference=StrictReference('exact', decide, basis='...'), fp_seeds=[...])`. A false positive is a disagreement with a stricter oracle, so without one there is no quantity here, only a distribution of scores.

silent-zero rate
RECORD_INCOMPLETE declined

the record this card was built from carries no recorded score trees

instrument the grader with `reward_lens.tap` so the per-leaf scores and the abstention channel are recorded, then pass `score_trees=` or `record_groups=` from the recorded groups. A composed score written down as one number has no channel to count, so this cannot be recovered from a record that did not keep one.

flakiness spread
The same policy on the same task scored 50.0% on every one of 3 identical runs across 240 task(s). This environment is deterministic here, so a measured delta is not competing with its spread.
exploit-family accounting
ABOVE_LOD_BELOW_LOQ detected, not quantifiable

f2 = 0: no family in this log of 259 finds appears exactly twice, so f1**2/(2*f2) is undefined. f1 = 1 over 4 families found.

bound: 0

collect about 259 more finds under the same search and re-run. The doubleton class is what Chao1 divides by, and an equiprobable-family model says a log this length should populate it after roughly that much more searching. The bias-corrected bound f1*(f1-1)/2 = 0.0 unseen families is attached and is a floor, not an estimate. Good-Turing's novelty probability f1/n = 0.004 needs no doubletons and is available now at rung 0.

variance components
0% of your score variance is which argument order you drew. %GRR = 0%, ndc = 2147483647. Every distinct value of the thing being scored is resolvable. This design has one observation per cell, so it carries no replication to estimate repeatability from and a measured zero here is the weakest possible evidence for it. Vary a facet and score again, or read the flakiness spread.
effective group size
RECORD_INCOMPLETE declined

the record this card was built from carries no per-group score vectors

pass the per-group score vectors the run actually produced: `CardInputs(group_scores=GroupScores.of([...]))`. The effective size is the nominal group size discounted by how much of the within-group spread is grader noise, so it is a property of recorded groups and not of the grader alone.

curl mass
SUBSTRATE_MISMATCH declined

this instrument applies to NEURAL_GEN, PROCEDURAL; the grader is PROGRAM

use an instrument declared for PROGRAM. A PROGRAM grader is a different kind of object, not a harder case of the same one.

Afriat index
SUBSTRATE_MISMATCH declined

this instrument applies to NEURAL_GEN, PROCEDURAL; the grader is PROGRAM

use an instrument declared for PROGRAM. A PROGRAM grader is a different kind of object, not a harder case of the same one.

abstention channel
RECORD_INCOMPLETE declined

the record this card was built from carries no recorded score trees

instrument the grader with `reward_lens.tap` so the per-leaf scores and the abstention channel are recorded, then pass `score_trees=` or `record_groups=` from the recorded groups. A composed score written down as one number has no channel to count, so this cannot be recovered from a record that did not keep one.

SWE-bench grading.py get_eval_tests_report

exploratory

grader SWE-bench Lite gold test lists, 240 rollouts

swebench/harness/grading.py (swebench 4.1.0, sha256 88fc500ebaf692a5)

reward-lens--github/reward-lens-assay/experiments/x1_release/cards/swebench.txt read reward-lens--github/reward-lens-assay/experiments/x1_release/cards/swebench.txt refused The lowest trust among them is exploratory.

coverage
Across 240 rollouts get_eval_tests_report has never taken 82% of its branches (33 of 40). Those are behaviours it cannot distinguish.
  • never fired: get_logs_eval.clause1, get_logs_eval.clause2, get_logs_eval.clause3, get_logs_eval.clause4, get_eval_tests_report.clause1, compute_fail_to_pass.clause1, compute_pass_to_pass.clause1, get_resolution_status.clause1, get_resolution_status.clause2, get_eval_report.clause1, get_eval_report.clause2, get_eval_report.clause3, get_eval_report.clause4
  • a random sample of the same size reaches 0.175 against the corpus's 0.175
surviving mutants
Of 60 mutants of get_eval_tests_report, 58 survive: 58 ways it could be wrong that no rollout in your corpus would reveal.
  • mutation score 0.033 over 60 competent
  • reproducers withheld. this payload enumerates concrete ways to make the grader wrong. It is written to the store with a sensitive flag, excluded from rendered artifacts unless explicitly requested, and not published without a recorded decision.
metamorphic violations
metamorphic violations on grade
  • 0 violations over 0 applicable rollout-relation pairs (nan%)
  • corpus 240 rollouts, 60 accepted at score > 0
  • baseline (identity transformation): 0 violations
  • (+5 more lines in this instrument's own report)
sensitivity profile
RECORD_INCOMPLETE declined

the record this card was built from carries no rubric scorer over named numeric inputs and no declared input ranges

expose the grader as a scorer over named numeric inputs and declare their ranges: `CardInputs(scorer=fn, rubric_inputs=[RubricInput('helpfulness', 0.0, 1.0), ...])`. A Sobol decomposition apportions output variance across an input space, so a grader with no numeric input space has no sensitivity profile rather than a flat one.

false-positive catalogue
RECORD_INCOMPLETE declined

the record this card was built from carries no stricter reference oracle and no seed rollouts to search around

supply a callable grader, a stricter reference that decides the same question, and seed rollouts to search around: `CardInputs(grader=fn, reference=StrictReference('exact', decide, basis='...'), fp_seeds=[...])`. A false positive is a disagreement with a stricter oracle, so without one there is no quantity here, only a distribution of scores.

silent-zero rate
RECORD_INCOMPLETE declined

the record this card was built from carries no recorded score trees

instrument the grader with `reward_lens.tap` so the per-leaf scores and the abstention channel are recorded, then pass `score_trees=` or `record_groups=` from the recorded groups. A composed score written down as one number has no channel to count, so this cannot be recovered from a record that did not keep one.

flakiness spread
The same policy on the same task scored 25.0% on every one of 3 identical runs across 240 task(s). This environment is deterministic here, so a measured delta is not competing with its spread.
exploit-family accounting
RECORD_INCOMPLETE declined

the record this card was built from carries no exploit log of finds by family

log every exploit you have already found, with its family and the effort it took, and pass it: `CardInputs(exploit_log=ExploitLog.of([ExploitFind(family=..., effort=...), ...]))`. The unseen mass is a Good-Turing estimate off the frequency spectrum of what has been found, so a blacklist with no counts on it cannot produce one and the fix is where the blacklist is kept.

variance components
40% of your score variance is evaluation type disagreement plus residual. %GRR = 89%, ndc = 0. This gauge cannot resolve two adjacent items: the measurement spread is as large as the spread it is measuring.
effective group size
RECORD_INCOMPLETE declined

the record this card was built from carries no per-group score vectors

pass the per-group score vectors the run actually produced: `CardInputs(group_scores=GroupScores.of([...]))`. The effective size is the nominal group size discounted by how much of the within-group spread is grader noise, so it is a property of recorded groups and not of the grader alone.

curl mass
SUBSTRATE_MISMATCH declined

this instrument applies to NEURAL_GEN, PROCEDURAL; the grader is PROGRAM

use an instrument declared for PROGRAM. A PROGRAM grader is a different kind of object, not a harder case of the same one.

Afriat index
SUBSTRATE_MISMATCH declined

this instrument applies to NEURAL_GEN, PROCEDURAL; the grader is PROGRAM

use an instrument declared for PROGRAM. A PROGRAM grader is a different kind of object, not a harder case of the same one.

abstention channel
RECORD_INCOMPLETE declined

the record this card was built from carries no recorded score trees

instrument the grader with `reward_lens.tap` so the per-leaf scores and the abstention channel are recorded, then pass `score_trees=` or `record_groups=` from the recorded groups. A composed score written down as one number has no channel to count, so this cannot be recovered from a record that did not keep one.

verl reward_score/gsm8k.py compute_score

exploratory

environment GSM8K test split, 240 rollouts

verl/utils/reward_score/gsm8k.py (sha256 497c6e32b708d0bc)

reward-lens--github/reward-lens-assay/experiments/x1_release/cards/verl_gsm8k.txt read reward-lens--github/reward-lens-assay/experiments/x1_release/cards/verl_gsm8k.txt refused The lowest trust among them is exploratory.

coverage
Across 240 rollouts compute_score has never taken 61% of its branches (11 of 18). Those are behaviours it cannot distinguish.
  • never fired: extract_solution.clause3, extract_solution.clause4, extract_solution.clause5, extract_solution.clause6, compute_score.clause1
  • a random sample of the same size reaches 0.389 against the corpus's 0.389
surviving mutants
Of 60 mutants of compute_score, 36 survive: 36 ways it could be wrong that no rollout in your corpus would reveal.
  • mutation score 0.400 over 60 competent
  • reproducers withheld. this payload enumerates concrete ways to make the grader wrong. It is written to the store with a sensitive flag, excluded from rendered artifacts unless explicitly requested, and not published without a recorded decision.
metamorphic violations
metamorphic violations on grade
  • 122 violations over 430 applicable rollout-relation pairs (28.37%)
  • corpus 240 rollouts, 122 accepted at score > 0
  • baseline (identity transformation): 0 violations
  • (+4 more lines in this instrument's own report)
sensitivity profile
RECORD_INCOMPLETE declined

the record this card was built from carries no rubric scorer over named numeric inputs and no declared input ranges

expose the grader as a scorer over named numeric inputs and declare their ranges: `CardInputs(scorer=fn, rubric_inputs=[RubricInput('helpfulness', 0.0, 1.0), ...])`. A Sobol decomposition apportions output variance across an input space, so a grader with no numeric input space has no sensitivity profile rather than a flat one.

false-positive catalogue
RECORD_INCOMPLETE declined

the record this card was built from carries no stricter reference oracle and no seed rollouts to search around

supply a callable grader, a stricter reference that decides the same question, and seed rollouts to search around: `CardInputs(grader=fn, reference=StrictReference('exact', decide, basis='...'), fp_seeds=[...])`. A false positive is a disagreement with a stricter oracle, so without one there is no quantity here, only a distribution of scores.

silent-zero rate
RECORD_INCOMPLETE declined

the record this card was built from carries no recorded score trees

instrument the grader with `reward_lens.tap` so the per-leaf scores and the abstention channel are recorded, then pass `score_trees=` or `record_groups=` from the recorded groups. A composed score written down as one number has no channel to count, so this cannot be recovered from a record that did not keep one.

flakiness spread
The same policy on the same task scored 50.8% on every one of 3 identical runs across 240 task(s). This environment is deterministic here, so a measured delta is not competing with its spread.
exploit-family accounting
You have found 1 exploit family in 122 rollout-relation trials-logged finds from verl gsm8k compute_score.
  • Chao1 puts 0.0% of the families still unseen [nan%, nan%] (0.00 unseen against 1 seen), and that is a floor rather than an estimate.
  • Good-Turing puts the probability that the next find is novel at 0.000 [0.000, 0.000] at 95%, by Esty's variance.
  • f1 = 0, f2 = 0, n = 122
  • (+4 more lines in this instrument's own report)
variance components
0% of your score variance is which extraction method you drew. %GRR = 0%, ndc = 2147483647. Every distinct value of the thing being scored is resolvable. This design has one observation per cell, so it carries no replication to estimate repeatability from and a measured zero here is the weakest possible evidence for it. Vary a facet and score again, or read the flakiness spread.
effective group size
RECORD_INCOMPLETE declined

the record this card was built from carries no per-group score vectors

pass the per-group score vectors the run actually produced: `CardInputs(group_scores=GroupScores.of([...]))`. The effective size is the nominal group size discounted by how much of the within-group spread is grader noise, so it is a property of recorded groups and not of the grader alone.

curl mass
SUBSTRATE_MISMATCH declined

this instrument applies to NEURAL_GEN, PROCEDURAL; the grader is PROGRAM

use an instrument declared for PROGRAM. A PROGRAM grader is a different kind of object, not a harder case of the same one.

Afriat index
SUBSTRATE_MISMATCH declined

this instrument applies to NEURAL_GEN, PROCEDURAL; the grader is PROGRAM

use an instrument declared for PROGRAM. A PROGRAM grader is a different kind of object, not a harder case of the same one.

abstention channel
RECORD_INCOMPLETE declined

the record this card was built from carries no recorded score trees

instrument the grader with `reward_lens.tap` so the per-leaf scores and the abstention channel are recorded, then pass `score_trees=` or `record_groups=` from the recorded groups. A composed score written down as one number has no channel to count, so this cannot be recovered from a record that did not keep one.

verl reward_score/search_r1_like_qa_em.py compute_score

exploratory

environment NQ-Open validation split, 240 rollouts

verl/utils/reward_score/search_r1_like_qa_em.py (sha256 8070e4d07719f23e)

reward-lens--github/reward-lens-assay/experiments/x1_release/cards/verl_search_r1.txt read reward-lens--github/reward-lens-assay/experiments/x1_release/cards/verl_search_r1.txt refused The lowest trust among them is exploratory.

coverage
Across 240 rollouts compute_score has never taken 57% of its branches (17 of 30). Those are behaviours it cannot distinguish.
  • never fired: em_check.clause1, subem_check.clause1, subem_check.clause2, extract_solution.clause1, compute_score.clause3, compute_score.clause5, compute_score_subem.clause1, compute_score_subem.clause2, compute_score_subem.clause3
  • a random sample of the same size reaches 0.433 against the corpus's 0.433
surviving mutants
Of 60 mutants of compute_score, 34 survive: 34 ways it could be wrong that no rollout in your corpus would reveal.
  • mutation score 0.433 over 60 competent
  • reproducers withheld. this payload enumerates concrete ways to make the grader wrong. It is written to the store with a sensitive flag, excluded from rendered artifacts unless explicitly requested, and not published without a recorded decision.
metamorphic violations
metamorphic violations on grade
  • 2 violations over 362 applicable rollout-relation pairs (0.55%)
  • corpus 240 rollouts, 120 accepted at score > 0
  • baseline (identity transformation): 0 violations
  • (+4 more lines in this instrument's own report)
sensitivity profile
RECORD_INCOMPLETE declined

the record this card was built from carries no rubric scorer over named numeric inputs and no declared input ranges

expose the grader as a scorer over named numeric inputs and declare their ranges: `CardInputs(scorer=fn, rubric_inputs=[RubricInput('helpfulness', 0.0, 1.0), ...])`. A Sobol decomposition apportions output variance across an input space, so a grader with no numeric input space has no sensitivity profile rather than a flat one.

false-positive catalogue
RECORD_INCOMPLETE declined

the record this card was built from carries no stricter reference oracle and no seed rollouts to search around

supply a callable grader, a stricter reference that decides the same question, and seed rollouts to search around: `CardInputs(grader=fn, reference=StrictReference('exact', decide, basis='...'), fp_seeds=[...])`. A false positive is a disagreement with a stricter oracle, so without one there is no quantity here, only a distribution of scores.

silent-zero rate
RECORD_INCOMPLETE declined

the record this card was built from carries no recorded score trees

instrument the grader with `reward_lens.tap` so the per-leaf scores and the abstention channel are recorded, then pass `score_trees=` or `record_groups=` from the recorded groups. A composed score written down as one number has no channel to count, so this cannot be recovered from a record that did not keep one.

flakiness spread
The same policy on the same task scored 50.0% on every one of 3 identical runs across 240 task(s). This environment is deterministic here, so a measured delta is not competing with its spread.
exploit-family accounting
You have found 1 exploit family in 2 rollout-relation trials-logged finds from verl search_r1 compute_score.
  • Chao1 puts 0.0% of the families still unseen [nan%, nan%] (0.00 unseen against 1 seen), and that is a floor rather than an estimate.
  • Good-Turing puts the probability that the next find is novel at 0.000 [0.000, 1.000] at 95%, by Esty's variance.
  • f1 = 0, f2 = 1, n = 2
  • (+4 more lines in this instrument's own report)
variance components
1% of your score variance is match rule disagreement plus residual. %GRR = 9%, ndc = 15. This gauge resolves at least 15 distinct levels of the thing being scored.
effective group size
RECORD_INCOMPLETE declined

the record this card was built from carries no per-group score vectors

pass the per-group score vectors the run actually produced: `CardInputs(group_scores=GroupScores.of([...]))`. The effective size is the nominal group size discounted by how much of the within-group spread is grader noise, so it is a property of recorded groups and not of the grader alone.

curl mass
SUBSTRATE_MISMATCH declined

this instrument applies to NEURAL_GEN, PROCEDURAL; the grader is PROGRAM

use an instrument declared for PROGRAM. A PROGRAM grader is a different kind of object, not a harder case of the same one.

Afriat index
SUBSTRATE_MISMATCH declined

this instrument applies to NEURAL_GEN, PROCEDURAL; the grader is PROGRAM

use an instrument declared for PROGRAM. A PROGRAM grader is a different kind of object, not a harder case of the same one.

abstention channel
RECORD_INCOMPLETE declined

the record this card was built from carries no recorded score trees

instrument the grader with `reward_lens.tap` so the per-leaf scores and the abstention channel are recorded, then pass `score_trees=` or `record_groups=` from the recorded groups. A composed score written down as one number has no channel to count, so this cannot be recovered from a record that did not keep one.

log_only

hendrycks/math is_equiv

exploratory

grader MATH-500 gold answers, 240 rollouts

https://raw.githubusercontent.com/hendrycks/math/main/modeling/math_equivalence.py (sha256 c4101b1f51a2bb65)

reward-lens--github/reward-lens-assay/experiments/x1_release/cards/is_equiv-log-only.txt read reward-lens--github/reward-lens-assay/experiments/x1_release/cards/is_equiv-log-only.txt refused Nothing read, so this card is a list of what it would take.

coverage
ACCESS_INSUFFICIENT declined

needs GRADER: SOURCE; you have GRADER: RECORD, RECORD: RECORD

supply grader at SOURCE. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.

surviving mutants
ACCESS_INSUFFICIENT declined

needs GRADER: SOURCE; you have GRADER: RECORD, RECORD: RECORD

supply grader at SOURCE. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.

metamorphic violations
ACCESS_INSUFFICIENT declined

needs GRADER: QUERY; you have GRADER: RECORD, RECORD: RECORD

supply grader at QUERY. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.

sensitivity profile
ACCESS_INSUFFICIENT declined

needs GRADER: QUERY; you have GRADER: RECORD, RECORD: RECORD

supply grader at QUERY. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.

false-positive catalogue
ACCESS_INSUFFICIENT declined

needs GOLD: QUERY, GRADER: QUERY; you have GRADER: RECORD, RECORD: RECORD

supply gold at QUERY; grader at QUERY. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.

silent-zero rate
RECORD_INCOMPLETE declined

the record this card was built from carries no recorded score trees

instrument the grader with `reward_lens.tap` so the per-leaf scores and the abstention channel are recorded, then pass `score_trees=` or `record_groups=` from the recorded groups. A composed score written down as one number has no channel to count, so this cannot be recovered from a record that did not keep one.

flakiness spread
ACCESS_INSUFFICIENT declined

needs TASK: QUERY|REPLICATE; you have GRADER: RECORD, RECORD: RECORD

supply task at QUERY; task at REPLICATE. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.

exploit-family accounting
ABOVE_LOD_BELOW_LOQ detected, not quantifiable

f2 = 0: no family in this log of 259 finds appears exactly twice, so f1**2/(2*f2) is undefined. f1 = 1 over 4 families found.

bound: 0

collect about 259 more finds under the same search and re-run. The doubleton class is what Chao1 divides by, and an equiprobable-family model says a log this length should populate it after roughly that much more searching. The bias-corrected bound f1*(f1-1)/2 = 0.0 unseen families is attached and is a floor, not an estimate. Good-Turing's novelty probability f1/n = 0.004 needs no doubletons and is available now at rung 0.

variance components
ACCESS_INSUFFICIENT declined

needs GRADER: REPLICATE; you have GRADER: RECORD, RECORD: RECORD

supply grader at REPLICATE. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.

effective group size
RECORD_INCOMPLETE declined

the record this card was built from carries no per-group score vectors

pass the per-group score vectors the run actually produced: `CardInputs(group_scores=GroupScores.of([...]))`. The effective size is the nominal group size discounted by how much of the within-group spread is grader noise, so it is a property of recorded groups and not of the grader alone.

curl mass
ACCESS_INSUFFICIENT declined

needs GRADER: QUERY; you have GRADER: RECORD, RECORD: RECORD

supply grader at QUERY. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.

Afriat index
SUBSTRATE_MISMATCH declined

this instrument applies to NEURAL_GEN, PROCEDURAL; the grader is PROGRAM

use an instrument declared for PROGRAM. A PROGRAM grader is a different kind of object, not a harder case of the same one.

abstention channel
RECORD_INCOMPLETE declined

the record this card was built from carries no recorded score trees

instrument the grader with `reward_lens.tap` so the per-leaf scores and the abstention channel are recorded, then pass `score_trees=` or `record_groups=` from the recorded groups. A composed score written down as one number has no channel to count, so this cannot be recovered from a record that did not keep one.

SWE-bench grading.py get_eval_tests_report

exploratory

grader SWE-bench Lite gold test lists, 240 rollouts

swebench/harness/grading.py (swebench 4.1.0, sha256 88fc500ebaf692a5)

reward-lens--github/reward-lens-assay/experiments/x1_release/cards/swebench-log-only.txt read reward-lens--github/reward-lens-assay/experiments/x1_release/cards/swebench-log-only.txt refused Nothing read, so this card is a list of what it would take.

coverage
ACCESS_INSUFFICIENT declined

needs GRADER: SOURCE; you have GRADER: RECORD, RECORD: RECORD

supply grader at SOURCE. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.

surviving mutants
ACCESS_INSUFFICIENT declined

needs GRADER: SOURCE; you have GRADER: RECORD, RECORD: RECORD

supply grader at SOURCE. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.

metamorphic violations
ACCESS_INSUFFICIENT declined

needs GRADER: QUERY; you have GRADER: RECORD, RECORD: RECORD

supply grader at QUERY. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.

sensitivity profile
ACCESS_INSUFFICIENT declined

needs GRADER: QUERY; you have GRADER: RECORD, RECORD: RECORD

supply grader at QUERY. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.

false-positive catalogue
ACCESS_INSUFFICIENT declined

needs GOLD: QUERY, GRADER: QUERY; you have GRADER: RECORD, RECORD: RECORD

supply gold at QUERY; grader at QUERY. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.

silent-zero rate
RECORD_INCOMPLETE declined

the record this card was built from carries no recorded score trees

instrument the grader with `reward_lens.tap` so the per-leaf scores and the abstention channel are recorded, then pass `score_trees=` or `record_groups=` from the recorded groups. A composed score written down as one number has no channel to count, so this cannot be recovered from a record that did not keep one.

flakiness spread
ACCESS_INSUFFICIENT declined

needs TASK: QUERY|REPLICATE; you have GRADER: RECORD, RECORD: RECORD

supply task at QUERY; task at REPLICATE. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.

exploit-family accounting
RECORD_INCOMPLETE declined

the record this card was built from carries no exploit log of finds by family

log every exploit you have already found, with its family and the effort it took, and pass it: `CardInputs(exploit_log=ExploitLog.of([ExploitFind(family=..., effort=...), ...]))`. The unseen mass is a Good-Turing estimate off the frequency spectrum of what has been found, so a blacklist with no counts on it cannot produce one and the fix is where the blacklist is kept.

variance components
ACCESS_INSUFFICIENT declined

needs GRADER: REPLICATE; you have GRADER: RECORD, RECORD: RECORD

supply grader at REPLICATE. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.

effective group size
RECORD_INCOMPLETE declined

the record this card was built from carries no per-group score vectors

pass the per-group score vectors the run actually produced: `CardInputs(group_scores=GroupScores.of([...]))`. The effective size is the nominal group size discounted by how much of the within-group spread is grader noise, so it is a property of recorded groups and not of the grader alone.

curl mass
ACCESS_INSUFFICIENT declined

needs GRADER: QUERY; you have GRADER: RECORD, RECORD: RECORD

supply grader at QUERY. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.

Afriat index
SUBSTRATE_MISMATCH declined

this instrument applies to NEURAL_GEN, PROCEDURAL; the grader is PROGRAM

use an instrument declared for PROGRAM. A PROGRAM grader is a different kind of object, not a harder case of the same one.

abstention channel
RECORD_INCOMPLETE declined

the record this card was built from carries no recorded score trees

instrument the grader with `reward_lens.tap` so the per-leaf scores and the abstention channel are recorded, then pass `score_trees=` or `record_groups=` from the recorded groups. A composed score written down as one number has no channel to count, so this cannot be recovered from a record that did not keep one.

verl reward_score/gsm8k.py compute_score

exploratory

environment GSM8K test split, 240 rollouts

verl/utils/reward_score/gsm8k.py (sha256 497c6e32b708d0bc)

reward-lens--github/reward-lens-assay/experiments/x1_release/cards/verl_gsm8k-log-only.txt read reward-lens--github/reward-lens-assay/experiments/x1_release/cards/verl_gsm8k-log-only.txt refused The lowest trust among them is exploratory.

coverage
ACCESS_INSUFFICIENT declined

needs GRADER: SOURCE; you have GRADER: RECORD, RECORD: RECORD

supply grader at SOURCE. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.

surviving mutants
ACCESS_INSUFFICIENT declined

needs GRADER: SOURCE; you have GRADER: RECORD, RECORD: RECORD

supply grader at SOURCE. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.

metamorphic violations
ACCESS_INSUFFICIENT declined

needs GRADER: QUERY; you have GRADER: RECORD, RECORD: RECORD

supply grader at QUERY. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.

sensitivity profile
ACCESS_INSUFFICIENT declined

needs GRADER: QUERY; you have GRADER: RECORD, RECORD: RECORD

supply grader at QUERY. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.

false-positive catalogue
ACCESS_INSUFFICIENT declined

needs GOLD: QUERY, GRADER: QUERY; you have GRADER: RECORD, RECORD: RECORD

supply gold at QUERY; grader at QUERY. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.

silent-zero rate
RECORD_INCOMPLETE declined

the record this card was built from carries no recorded score trees

instrument the grader with `reward_lens.tap` so the per-leaf scores and the abstention channel are recorded, then pass `score_trees=` or `record_groups=` from the recorded groups. A composed score written down as one number has no channel to count, so this cannot be recovered from a record that did not keep one.

flakiness spread
ACCESS_INSUFFICIENT declined

needs TASK: QUERY|REPLICATE; you have GRADER: RECORD, RECORD: RECORD

supply task at QUERY; task at REPLICATE. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.

exploit-family accounting
You have found 1 exploit family in 122 rollout-relation trials-logged finds from verl gsm8k compute_score.
  • Chao1 puts 0.0% of the families still unseen [nan%, nan%] (0.00 unseen against 1 seen), and that is a floor rather than an estimate.
  • Good-Turing puts the probability that the next find is novel at 0.000 [0.000, 0.000] at 95%, by Esty's variance.
  • f1 = 0, f2 = 0, n = 122
  • (+4 more lines in this instrument's own report)
variance components
ACCESS_INSUFFICIENT declined

needs GRADER: REPLICATE; you have GRADER: RECORD, RECORD: RECORD

supply grader at REPLICATE. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.

effective group size
RECORD_INCOMPLETE declined

the record this card was built from carries no per-group score vectors

pass the per-group score vectors the run actually produced: `CardInputs(group_scores=GroupScores.of([...]))`. The effective size is the nominal group size discounted by how much of the within-group spread is grader noise, so it is a property of recorded groups and not of the grader alone.

curl mass
ACCESS_INSUFFICIENT declined

needs GRADER: QUERY; you have GRADER: RECORD, RECORD: RECORD

supply grader at QUERY. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.

Afriat index
SUBSTRATE_MISMATCH declined

this instrument applies to NEURAL_GEN, PROCEDURAL; the grader is PROGRAM

use an instrument declared for PROGRAM. A PROGRAM grader is a different kind of object, not a harder case of the same one.

abstention channel
RECORD_INCOMPLETE declined

the record this card was built from carries no recorded score trees

instrument the grader with `reward_lens.tap` so the per-leaf scores and the abstention channel are recorded, then pass `score_trees=` or `record_groups=` from the recorded groups. A composed score written down as one number has no channel to count, so this cannot be recovered from a record that did not keep one.

verl reward_score/search_r1_like_qa_em.py compute_score

exploratory

environment NQ-Open validation split, 240 rollouts

verl/utils/reward_score/search_r1_like_qa_em.py (sha256 8070e4d07719f23e)

reward-lens--github/reward-lens-assay/experiments/x1_release/cards/verl_search_r1-log-only.txt read reward-lens--github/reward-lens-assay/experiments/x1_release/cards/verl_search_r1-log-only.txt refused The lowest trust among them is exploratory.

coverage
ACCESS_INSUFFICIENT declined

needs GRADER: SOURCE; you have GRADER: RECORD, RECORD: RECORD

supply grader at SOURCE. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.

surviving mutants
ACCESS_INSUFFICIENT declined

needs GRADER: SOURCE; you have GRADER: RECORD, RECORD: RECORD

supply grader at SOURCE. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.

metamorphic violations
ACCESS_INSUFFICIENT declined

needs GRADER: QUERY; you have GRADER: RECORD, RECORD: RECORD

supply grader at QUERY. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.

sensitivity profile
ACCESS_INSUFFICIENT declined

needs GRADER: QUERY; you have GRADER: RECORD, RECORD: RECORD

supply grader at QUERY. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.

false-positive catalogue
ACCESS_INSUFFICIENT declined

needs GOLD: QUERY, GRADER: QUERY; you have GRADER: RECORD, RECORD: RECORD

supply gold at QUERY; grader at QUERY. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.

silent-zero rate
RECORD_INCOMPLETE declined

the record this card was built from carries no recorded score trees

instrument the grader with `reward_lens.tap` so the per-leaf scores and the abstention channel are recorded, then pass `score_trees=` or `record_groups=` from the recorded groups. A composed score written down as one number has no channel to count, so this cannot be recovered from a record that did not keep one.

flakiness spread
ACCESS_INSUFFICIENT declined

needs TASK: QUERY|REPLICATE; you have GRADER: RECORD, RECORD: RECORD

supply task at QUERY; task at REPLICATE. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.

exploit-family accounting
You have found 1 exploit family in 2 rollout-relation trials-logged finds from verl search_r1 compute_score.
  • Chao1 puts 0.0% of the families still unseen [nan%, nan%] (0.00 unseen against 1 seen), and that is a floor rather than an estimate.
  • Good-Turing puts the probability that the next find is novel at 0.000 [0.000, 1.000] at 95%, by Esty's variance.
  • f1 = 0, f2 = 1, n = 2
  • (+4 more lines in this instrument's own report)
variance components
ACCESS_INSUFFICIENT declined

needs GRADER: REPLICATE; you have GRADER: RECORD, RECORD: RECORD

supply grader at REPLICATE. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.

effective group size
RECORD_INCOMPLETE declined

the record this card was built from carries no per-group score vectors

pass the per-group score vectors the run actually produced: `CardInputs(group_scores=GroupScores.of([...]))`. The effective size is the nominal group size discounted by how much of the within-group spread is grader noise, so it is a property of recorded groups and not of the grader alone.

curl mass
ACCESS_INSUFFICIENT declined

needs GRADER: QUERY; you have GRADER: RECORD, RECORD: RECORD

supply grader at QUERY. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.

Afriat index
SUBSTRATE_MISMATCH declined

this instrument applies to NEURAL_GEN, PROCEDURAL; the grader is PROGRAM

use an instrument declared for PROGRAM. A PROGRAM grader is a different kind of object, not a harder case of the same one.

abstention channel
RECORD_INCOMPLETE declined

the record this card was built from carries no recorded score trees

instrument the grader with `reward_lens.tap` so the per-leaf scores and the abstention channel are recorded, then pass `score_trees=` or `record_groups=` from the recorded groups. A composed score written down as one number has no channel to count, so this cannot be recovered from a record that did not keep one.

The same four subjects, for a reader who holds only a log.

Two of the four cards drop to reward-lens--github/reward-lens-assay/experiments/x1_release/cards/is_equiv-log-only.txt fields read. The other two keep exactly one, the single field countable from a log alone. Nothing else survives, so a card is a list of what it would take.

The same four subjects, for a reader who holds only a log.
hendrycks/math is_equiv reward-lens--github/reward-lens-assay/experiments/x1_release/cards/is_equiv-log-only.txt reward-lens--github/reward-lens-assay/experiments/x1_release/cards/is_equiv-log-only.txt
SWE-bench grading.py reward-lens--github/reward-lens-assay/experiments/x1_release/cards/swebench-log-only.txt reward-lens--github/reward-lens-assay/experiments/x1_release/cards/swebench-log-only.txt
verl GSM8K reward reward-lens--github/reward-lens-assay/experiments/x1_release/cards/verl_gsm8k-log-only.txt reward-lens--github/reward-lens-assay/experiments/x1_release/cards/verl_gsm8k-log-only.txt
verl Search-R1 QA exact match reward-lens--github/reward-lens-assay/experiments/x1_release/cards/verl_search_r1-log-only.txt reward-lens--github/reward-lens-assay/experiments/x1_release/cards/verl_search_r1-log-only.txt

refusal reasons

where is the remedy answerable?

3 The mapping exists for three reasons and for no others. The library assigns no group to the other fourteen, so neither does this file.

  • ACCESS_INSUFFICIENT

    No estimator for this quantity works at the access you have. Silent degradation to a worse one is how a number becomes uninterpretable, so nothing was computed.

    Ask what the cheaper rung would cost. Every quantity has a ladder, and the refusal carries the rung that would work and what it needs, so `what_would_it_take` turns this into a shopping list: one more checkpoint, or logprobs on the sampling policy, or permission to call the grader twice on the same input. If none of that is available, the quantity is out of reach on this run and the honest move is to say so in the write-up rather than substitute something adjacent.

    remedy get more access, or drop a rung and accept its stated bias answerable where you are standing

  • RECORD_INCOMPLETE

    Your access is sufficient and the record does not carry the field this estimator reads. Nothing more can be recovered from this record; the fix is upstream, where it was written.

    Do not go looking for more access; it will not help. The field was never written, so nothing you do to this record recovers it. Fix it upstream: turn on the dump in whatever produced the run, or record the missing field on the next run. The refusal names both the field and what it is missing from, so "no `logprobs_sampling` on 412 of 512 trajectories" tells you whether this is a configuration problem or a partial write.

    remedy write the field and run again answerable upstream, where the record was produced

  • SUBSTRATE_MISMATCH

    This instrument does not apply to this kind of grader. A program has no activations; that is a category error rather than a hard case.

    Nothing to fix. You asked a question that does not apply to this kind of grader, most often an activation question of a program. Reach for the instrument built for this substrate: the capability report lists them, and for a program the verifier series answers the structural questions the white-box series answers for a network.

  • PHASE_MISMATCH

    This is an in-run question and the run is over, or a pre-run question and it has started.

    Either the run is over and you asked an in-run question, or it has not started and you asked a post-run one. If the run is over, the question has to be answered from the record, and the instrument that does that is a different one. If it has not started, this is the moment to record what the in-run instrument will need, because a phase you have passed cannot be revisited.

  • ENVELOPE_VIOLATED

    The estimator's assumptions do not hold on this run. An instrument that is available and invalid is worse than one that is unavailable.

    Read which condition failed and what its statistic was; the refusal carries both. Then pick one of three: restrict the analysis to a window where the condition holds, switch to a rung whose envelope does not require it, or accept that the quantity is not estimable on this run. An instrument that is available and invalid is worse than one that is unavailable, so the option that is not on the list is running it anyway.

  • BELOW_LOD

    The effect is smaller than the measurement substrate's disagreement with itself, so it is not attributable to the thing being measured.

    The effect is smaller than the measurement's disagreement with itself, so there is nothing here to interpret in either direction. This is not a negative result and must not be written up as one. To go further you need a smaller limit of detection: more replicates, a lower-variance readout, or a stimulus set that separates the conditions more sharply. The refusal carries the limit, so you can compute how much more.

  • ABOVE_LOD_BELOW_LOQ

    Detected but not quantifiable. A bound is returned; a point estimate would be false precision.

    Use the bound. It is real, it is in `partial`, and an upper bound is a usable answer for most decisions a point estimate would have been used for. Report it as a bound and say so. If the decision genuinely needs the point estimate, the limit of quantification tells you how much more data would get you there.

  • ESS_BELOW_FLOOR

    The importance weights have degenerated, so this is past the visibility horizon and any number would be a guess wearing an interval.

    The importance weights have degenerated, so you are extrapolating past the point where the data constrains anything. Shorten the extrapolation: ask for the quantity at a smaller distance from the sampling policy, where the effective sample size is still above the floor. Widening the interval instead is the mistake this refusal exists to prevent, because past the horizon the interval is not wide, it is undefined.

  • NO_MATCHED_CONTROL

    A null with no identically-powered positive control cannot be distinguished from an underpowered experiment.

    You asked for a null and there is no positive control at the same power, so a real absence and an underpowered experiment look identical. Run the matched positive control from `stats/baselines`: it is a case where the effect is known to exist, at the same n and the same readout. If the control also comes back null, the experiment is underpowered and the number of samples that would fix it is a power calculation away.

  • GAUGE_MISMATCH

    A covariant quantity was compared across frames with no shared basis, so the difference would be a coordinate artifact.

    You compared a covariant quantity across two frames with no shared basis, which makes the difference a coordinate artifact rather than a finding. Fit a shared frame and compare in it, or compare an invariant of the two quantities instead. Comparing raw coordinates across models is the specific error this gate exists to catch, and it produces numbers that look reasonable, which is why it needs a gate rather than a warning.

  • UNIT_MISMATCH

    Two quantities in incompatible units were compared. The conversion factor is a property of the data, not of the unit, so this is not converted silently.

    Two quantities in incompatible units met, most often per-token against per-sequence. Decide which unit the question is actually in and get both sides into it, which needs data the comparison did not have: how many tokens, or how many sequences. The library will not do that conversion for you, because the factor is a property of your data rather than of the unit.

  • REFERENCE_UNCERTIFIED

    The reference material carries no uncertainty of its own. You cannot calibrate against an uncalibrated ruler.

    The reference material you calibrated against has no uncertainty of its own, so the calibrated number would inherit an error nobody has measured. Either certify the reference, which means measuring its own uncertainty and recording it, or use a certified one. If neither is possible, report the reading as uncalibrated and let it sit at the trust level that implies.

  • LABEL_QUALITY_UNKNOWN

    The labels have no measured error rate, so scoring against them measures the labels.

    Scoring against labels whose error rate nobody measured measures the labels. Get a measured error rate: a doubly-labelled subset is usually enough, and the label metrology series exists to turn one into an error rate with an interval. Until then any score against these labels is bounded above by their quality and the bound is unknown.

  • PLAN_NOT_CLOSED

    A prediction names a metric that no arc in this plan produces. Found before anything ran.

    A prediction names a metric that no arc in this plan produces, so the prediction could never have been graded. Either add the arc that produces the metric or change the prediction to one the plan answers. This fires before anything runs, which is the only useful time for it to fire.

  • BUDGET_EXCEEDED

    The costed plan exceeds the declared budget.

    The costed plan is more expensive than the budget you declared. Cut the plan, raise the budget deliberately, or drop to a cheaper rung and accept its bias, which the ladder states. The one thing not to do is run it and find out, because a plan that runs out of budget half way produces an arc nobody can interpret.

  • VOID

    The run is not readable, which is different from a negative result.

    The run is not readable, which is a different thing from a negative result and must never be written up as one. Find out why: a truncated write, a missing manifest, a schema the reader does not recognise. A void run contributes nothing in either direction, and counting it as evidence of no effect is how a broken pipeline becomes a published null.

  • QUANTITY_UNDEFINED

    This quantity is not defined for this object, so there is nothing here to measure at any access and from any record. The remedy names the question that does apply instead.

    Nothing to fix and nothing to buy: this question does not apply to this object, so no amount of access and no rewriting of the record will produce an answer. The commonest case is asking a mean-centred estimator for its amplification, when the amplification is a property of dividing by the group standard deviation and this estimator does not do that. Read the remedy, which names the question that **does** apply to what you have, and ask that one instead. If the named alternative is not the question you wanted, the honest conclusion is that the thing you wanted to know is not a property of this object, which is worth writing down rather than working around.

    remedy ask the question that does apply answerable nowhere

More than half the fields refused, and the one that read found a problem

Across four cards, more than half the fields refused.

reward-lens--github/reward-lens-assay/experiments/x1_release/cards/is_equiv.txt fields rendered, reward-lens--github/reward-lens-assay/experiments/x1_release/cards/is_equiv.txt read, reward-lens--github/reward-lens-assay/experiments/x1_release/cards/is_equiv.txt refused, and none of them for want of access.

Across four cards, more than half the fields refused.
hendrycks/math is_equiv reward-lens--github/reward-lens-assay/experiments/x1_release/cards/is_equiv.txt reward-lens--github/reward-lens-assay/experiments/x1_release/cards/is_equiv.txt
SWE-bench grading.py reward-lens--github/reward-lens-assay/experiments/x1_release/cards/swebench.txt reward-lens--github/reward-lens-assay/experiments/x1_release/cards/swebench.txt
verl GSM8K reward reward-lens--github/reward-lens-assay/experiments/x1_release/cards/verl_gsm8k.txt reward-lens--github/reward-lens-assay/experiments/x1_release/cards/verl_gsm8k.txt
verl Search-R1 QA exact match reward-lens--github/reward-lens-assay/experiments/x1_release/cards/verl_search_r1.txt reward-lens--github/reward-lens-assay/experiments/x1_release/cards/verl_search_r1.txt

The sharp finding.

The mathematical equivalence checker was mutated reward-lens--github/reward-lens-assay/experiments/x1_release/cards/is_equiv.txt ways. Mutation score reward-lens--github/reward-lens-assay/experiments/x1_release/cards/is_equiv.txt.

That checker ends in reward-lens--github/reward-lens-assay/experiments/x1_release/cards/is_equiv.txt bare exception handlers in about reward-lens--github/reward-lens-assay/experiments/x1_release/cards/is_equiv.txt lines, each converting an internal failure into a raw string comparison returned as a verdict. No caller can distinguish a real match from a fallback.

Under transformations no human marker would distinguish, its verdict changes on reward-lens--github/reward-lens-assay/experiments/x1_release/cards/is_equiv.txt of reward-lens--github/reward-lens-assay/experiments/x1_release/cards/is_equiv.txt applicable pairs.

A score of zero is an absence of detection on one checker against one corpus, at the denominator stated beside it. It is not a defect rate for verifiers in general and it is not an estimate of any population.

Fig. 3.2 · Sixty mutations, and the answer key that noticed none of them One checker, and the comparison rests on two of the other three subjects rather than all three: the same instrument kills a substantial share of mutants on both verl environments and almost none on SWE-bench's report builder. That is what makes a score of zero a finding rather than a broken harness. The counts are in the table.
Sixty mutations, and the answer key that noticed none of them
hendrycks/math is_equiv reward-lens--github/reward-lens-assay/experiments/x1_release/cards/is_equiv.txt reward-lens--github/reward-lens-assay/experiments/x1_release/cards/is_equiv.txt
SWE-bench grading.py reward-lens--github/reward-lens-assay/experiments/x1_release/cards/swebench.txt reward-lens--github/reward-lens-assay/experiments/x1_release/cards/swebench.txt
verl GSM8K reward reward-lens--github/reward-lens-assay/experiments/x1_release/cards/verl_gsm8k.txt reward-lens--github/reward-lens-assay/experiments/x1_release/cards/verl_gsm8k.txt
verl Search-R1 QA exact match reward-lens--github/reward-lens-assay/experiments/x1_release/cards/verl_search_r1.txt reward-lens--github/reward-lens-assay/experiments/x1_release/cards/verl_search_r1.txt

Each mark is one mutant of the equivalence checker, run against its own benchmark's gold answer pairs. A mutant is killed when the pairs disagree with the original. None were.

What this does not establish

None of the four subjects is a rollout from a policy under training. This says what a grader does to inputs, not what it does to an optimisation run.

None of the four subjects is a rollout from a policy under training
hendrycks/math is_equiv reward-lens--github/reward-lens-assay/experiments/x1_release/cards/is_equiv.txt reward-lens--github/reward-lens-assay/experiments/x1_release/cards/is_equiv.txt
SWE-bench grading.py reward-lens--github/reward-lens-assay/experiments/x1_release/cards/swebench.txt reward-lens--github/reward-lens-assay/experiments/x1_release/cards/swebench.txt
verl GSM8K reward reward-lens--github/reward-lens-assay/experiments/x1_release/cards/verl_gsm8k.txt reward-lens--github/reward-lens-assay/experiments/x1_release/cards/verl_gsm8k.txt
verl Search-R1 QA exact match reward-lens--github/reward-lens-assay/experiments/x1_release/cards/verl_search_r1.txt reward-lens--github/reward-lens-assay/experiments/x1_release/cards/verl_search_r1.txt

Under transformations no human marker would distinguish

Under transformations no human marker would distinguish
hendrycks/math is_equiv reward-lens--github/reward-lens-assay/experiments/x1_release/cards/is_equiv.txt reward-lens--github/reward-lens-assay/experiments/x1_release/cards/is_equiv.txt
SWE-bench grading.py reward-lens--github/reward-lens-assay/experiments/x1_release/cards/swebench.txt reward-lens--github/reward-lens-assay/experiments/x1_release/cards/swebench.txt
verl GSM8K reward reward-lens--github/reward-lens-assay/experiments/x1_release/cards/verl_gsm8k.txt reward-lens--github/reward-lens-assay/experiments/x1_release/cards/verl_gsm8k.txt
verl Search-R1 QA exact match reward-lens--github/reward-lens-assay/experiments/x1_release/cards/verl_search_r1.txt reward-lens--github/reward-lens-assay/experiments/x1_release/cards/verl_search_r1.txt

Why the refused fields refused, which bounds what a refusal establishes.

A refusal naming an incomplete record says something about the record the instrument was handed and nothing about the grader. Most of them are that one, so the argument above survives and is narrower than it first reads.

Why the refused fields refused, which bounds what a refusal establishes.
ABOVE_LOD_BELOW_LOQ reward-lens--github/reward-lens-assay/experiments/x1_release/cards/is_equiv.txt
RECORD_INCOMPLETE reward-lens--github/reward-lens-assay/experiments/x1_release/cards/is_equiv.txt
SUBSTRATE_MISMATCH reward-lens--github/reward-lens-assay/experiments/x1_release/cards/is_equiv.txt

What each rung licenses

The top rung of the trust ladder is defined, enforced, documented, and empty. Nothing in this repository has reached it.

This project scored its own calibration, and failed.

The earlier research log's directional Brier score was this project's audit of its own calibration refuted frozen before the data brier_directional 0.26 interval_coverage_0p8 0.75 n_directional 16 n_intervals 4 f93f4b5578c221d3d2665f9b305f3e86759be862+dirty Open the record › against a coin at this project's audit of its own calibration refuted frozen before the data brier_directional 0.26 interval_coverage_0p8 0.75 n_directional 16 n_intervals 4 f93f4b5578c221d3d2665f9b305f3e86759be862+dirty Open the record › . Its interval coverage was this project's audit of its own calibration refuted frozen before the data brier_directional 0.26 interval_coverage_0p8 0.75 n_directional 16 n_intervals 4 f93f4b5578c221d3d2665f9b305f3e86759be862+dirty Open the record › against a threshold of this project's audit of its own calibration refuted frozen before the data brier_directional 0.26 interval_coverage_0p8 0.75 n_directional 16 n_intervals 4 f93f4b5578c221d3d2665f9b305f3e86759be862+dirty Open the record › fixed before the data. The result contradicted its own prediction about itself. The recorded conclusion is that those directional calls do not beat a coin, and that the discipline does not calibrate itself.

A second one, because it is the better demonstration: a replication harness built here took a published row from this project's own ledger, tried to reproduce it from the evidence store, and reported that it did not.

Run it yourself

bash 
unzip grader.zip -d grader && python grader/reproduce.py

Reproduce this

Everything on this page regenerates from the bundle below.

contents
cards/is_equiv-log-only.txt, cards/is_equiv.txt, cards/swebench-log-only.txt, cards/swebench.txt, cards/verl_gsm8k-log-only.txt, cards/verl_gsm8k.txt, cards/verl_search_r1-log-only.txt, cards/verl_search_r1.txt, evidence.jsonl, refusals/is_equiv.json, refusals/swebench.json, refusals/verl_gsm8k.json, refusals/verl_search_r1.json, reproduce.py
size
public/bundles/grader.zip
sha256
4cc0f432a37baf218b0e2b5cf5348f8dd1a6383ad4f60675746e15dabba62a30
commit
public/bundles/grader.zip
needs
Python 3.10 or newer. No GPU, no model, no grader call and no network. Every field on every card re-derives from the artifacts in the bundle using the standard library alone. Pointing the same instrument at a scoring program you already have needs the installed library, and that is the next step rather than this one.

Download the data

Expected output

subject           access     read  refused  remedies  reasons
is_equiv          full          5        8         8  3
is_equiv          log_only      0       13        13  4
swebench          full          5        8         8  2
swebench          log_only      0       13        13  3
verl_gsm8k        full          6        7         7  2
verl_gsm8k        log_only      1       12        12  3
verl_search_r1    full          6        7         7  2
verl_search_r1    log_only      1       12        12  3

full       52 fields, 22 read, 30 refused
             ABOVE_LOD_BELOW_LOQ      1
             RECORD_INCOMPLETE        21
             SUBSTRATE_MISMATCH       8
log_only   52 fields, 2 read, 50 refused
             ABOVE_LOD_BELOW_LOQ      1
             ACCESS_INSUFFICIENT      32
             RECORD_INCOMPLETE        13
             SUBSTRATE_MISMATCH       4

structured refusal records  30
refusals parsed at full access  30
the two agree               yes

No independent reproduction recorded.