# verl reward_score/gsm8k.py compute_score
# kind      environment
# source    /home/suhail-nadaf/final-reward/assay-recon/frameworks/verl/verl/utils/reward_score/gsm8k.py (sha256 497c6e32b708d0bc)
# entry     compute_score
# corpus    GSM8K test split, 240 rollouts
# note      The facet is `method`, which `compute_score` takes as an argument and which verl's config sets: `strict` requires the `#### ` marker and `flexible` takes the last number anywhere in the completion. Both are shipped and both are used.
#
GRADER CARD  verl GSM8K reward
  what a buyer gets before spending money on a grader, and what an auditor gets after

  subject        verifier:59fbd42f0d190815b64ea96433d1d6ef
  substrate      PROGRAM    phase  PRE_RUN
  access         GOLD: QUERY, GRADER: RECORD|QUERY|REPLICATE|SOURCE, RECORD: RECORD, TASK: QUERY|REPLICATE|SOURCE
                 meets D7's stated minimum (GRADER: QUERY|REPLICATE)
  trust          exploratory, computed by the gates
  lowest reading exploratory
  envelope       STATIONARY_GRADER was not measured, so this check did not run. A card assembled across a grader edit describes two programs rather than one. D10 (`env.replay_fidelity`) is what measures whether it was one; pass its ConditionReading as `regime_reading` to have this checked.
  cost           2,400 grader calls, which is a floor: 5 of the 6 fields that read do not model their own cost
  not checked    envelope (regime not measured) on 5 field(s) and the card itself; limit of detection on 6 field(s) and the card itself

  6 of 13 fields read and 7 refused. The lowest trust among them is exploratory.

  coverage                   Across 240 rollouts compute_score has never taken 61% of its branches (11 of 18). Those are behaviours it cannot distinguish.  [exploratory]
                             never fired: extract_solution.clause3, extract_solution.clause4, extract_solution.clause5, extract_solution.clause6, compute_score.clause1
                             a random sample of the same size reaches 0.389 against the corpus's 0.389
  surviving mutants          Of 60 mutants of compute_score, 36 survive: 36 ways it could be wrong that no rollout in your corpus would reveal.  [exploratory]
                             mutation score 0.400 over 60 competent
                             reproducers withheld. this payload enumerates concrete ways to make the grader wrong. It is written to the store with a sensitive flag, excluded from rendered artifacts unless explicitly requested, and not published without a recorded decision.
  metamorphic violations     metamorphic violations on grade  [exploratory]
                             122 violations over 430 applicable rollout-relation pairs (28.37%)
                             corpus 240 rollouts, 122 accepted at score > 0
                             baseline (identity transformation): 0 violations
                             (+4 more lines in this instrument's own report)
  sensitivity profile        REFUSED  RECORD_INCOMPLETE
                             the record this card was built from carries no rubric scorer over named numeric inputs and no declared input ranges
                             Remedy: expose the grader as a scorer over named numeric inputs and declare their ranges: `CardInputs(scorer=fn, rubric_inputs=[RubricInput('helpfulness', 0.0, 1.0), ...])`. A Sobol decomposition apportions output variance across an input space, so a grader with no numeric input space has no sensitivity profile rather than a flat one.
  false-positive catalogue   REFUSED  RECORD_INCOMPLETE
                             the record this card was built from carries no stricter reference oracle and no seed rollouts to search around
                             Remedy: supply a callable grader, a stricter reference that decides the same question, and seed rollouts to search around: `CardInputs(grader=fn, reference=StrictReference('exact', decide, basis='...'), fp_seeds=[...])`. A false positive is a disagreement with a stricter oracle, so without one there is no quantity here, only a distribution of scores.
  silent-zero rate           REFUSED  RECORD_INCOMPLETE
                             the record this card was built from carries no recorded score trees
                             Remedy: instrument the grader with `reward_lens.tap` so the per-leaf scores and the abstention channel are recorded, then pass `score_trees=` or `record_groups=` from the recorded groups. A composed score written down as one number has no channel to count, so this cannot be recovered from a record that did not keep one.
  flakiness spread           The same policy on the same task scored 50.8% on every one of 3 identical runs across 240 task(s). This environment is deterministic here, so a measured delta is not competing with its spread.  [exploratory]
  exploit-family accounting  You have found 1 exploit family in 122 rollout-relation trials-logged finds from verl gsm8k compute_score.  [exploratory]
                             Chao1 puts 0.0% of the families still unseen [nan%, nan%] (0.00 unseen against 1 seen), and that is a floor rather than an estimate.
                             Good-Turing puts the probability that the next find is novel at 0.000 [0.000, 0.000] at 95%, by Esty's variance.
                             f1 = 0, f2 = 0, n = 122
                             (+4 more lines in this instrument's own report)
  variance components        0% of your score variance is which extraction method you drew. %GRR = 0%, ndc = 2147483647. Every distinct value of the thing being scored is resolvable. This design has one observation per cell, so it carries no replication to estimate repeatability from and a measured zero here is the weakest possible evidence for it. Vary a facet and score again, or read the flakiness spread.  [exploratory]
  effective group size       REFUSED  RECORD_INCOMPLETE
                             the record this card was built from carries no per-group score vectors
                             Remedy: pass the per-group score vectors the run actually produced: `CardInputs(group_scores=GroupScores.of([...]))`. The effective size is the nominal group size discounted by how much of the within-group spread is grader noise, so it is a property of recorded groups and not of the grader alone.
  curl mass                  REFUSED  SUBSTRATE_MISMATCH
                             this instrument applies to NEURAL_GEN, PROCEDURAL; the grader is PROGRAM
                             Remedy: use an instrument declared for PROGRAM. A PROGRAM grader is a different kind of object, not a harder case of the same one.
  Afriat index               REFUSED  SUBSTRATE_MISMATCH
                             this instrument applies to NEURAL_GEN, PROCEDURAL; the grader is PROGRAM
                             Remedy: use an instrument declared for PROGRAM. A PROGRAM grader is a different kind of object, not a harder case of the same one.
  abstention channel         REFUSED  RECORD_INCOMPLETE
                             the record this card was built from carries no recorded score trees
                             Remedy: instrument the grader with `reward_lens.tap` so the per-leaf scores and the abstention channel are recorded, then pass `score_trees=` or `record_groups=` from the recorded groups. A composed score written down as one number has no channel to count, so this cannot be recovered from a record that did not keep one.

  1 field(s) withheld their reproducers: surviving mutants. Pass include_sensitive=True with a recorded DisclosureDecision on the payload to see them.
