# verl reward_score/gsm8k.py compute_score
# kind      environment
# source    /home/suhail-nadaf/final-reward/assay-recon/frameworks/verl/verl/utils/reward_score/gsm8k.py (sha256 497c6e32b708d0bc)
# entry     compute_score
# corpus    GSM8K test split, 240 rollouts
# note      The facet is `method`, which `compute_score` takes as an argument and which verl's config sets: `strict` requires the `#### ` marker and `flexible` takes the last number anywhere in the completion. Both are shipped and both are used.
#
GRADER CARD  verl GSM8K reward
  what a buyer gets before spending money on a grader, and what an auditor gets after

  subject        verifier:59fbd42f0d190815b64ea96433d1d6ef
  substrate      PROGRAM    phase  PRE_RUN
  access         GRADER: RECORD, RECORD: RECORD
                 short of D7's stated minimum (GRADER: QUERY|REPLICATE) by GRADER: QUERY|REPLICATE
  trust          exploratory, computed by the gates
  lowest reading exploratory
  envelope       STATIONARY_GRADER was not measured, so this check did not run. A card assembled across a grader edit describes two programs rather than one. D10 (`env.replay_fidelity`) is what measures whether it was one; pass its ConditionReading as `regime_reading` to have this checked.
  cost           no field priced itself, which is a floor: 1 of the 1 fields that read do not model their own cost
  not checked    envelope (regime not measured) on 1 field(s) and the card itself; limit of detection on 1 field(s) and the card itself

  1 of 13 fields read and 12 refused. The lowest trust among them is exploratory.

  coverage                   REFUSED  ACCESS_INSUFFICIENT
                             needs GRADER: SOURCE; you have GRADER: RECORD, RECORD: RECORD
                             Remedy: supply grader at SOURCE. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.
  surviving mutants          REFUSED  ACCESS_INSUFFICIENT
                             needs GRADER: SOURCE; you have GRADER: RECORD, RECORD: RECORD
                             Remedy: supply grader at SOURCE. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.
  metamorphic violations     REFUSED  ACCESS_INSUFFICIENT
                             needs GRADER: QUERY; you have GRADER: RECORD, RECORD: RECORD
                             Remedy: supply grader at QUERY. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.
  sensitivity profile        REFUSED  ACCESS_INSUFFICIENT
                             needs GRADER: QUERY; you have GRADER: RECORD, RECORD: RECORD
                             Remedy: supply grader at QUERY. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.
  false-positive catalogue   REFUSED  ACCESS_INSUFFICIENT
                             needs GOLD: QUERY, GRADER: QUERY; you have GRADER: RECORD, RECORD: RECORD
                             Remedy: supply gold at QUERY; grader at QUERY. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.
  silent-zero rate           REFUSED  RECORD_INCOMPLETE
                             the record this card was built from carries no recorded score trees
                             Remedy: instrument the grader with `reward_lens.tap` so the per-leaf scores and the abstention channel are recorded, then pass `score_trees=` or `record_groups=` from the recorded groups. A composed score written down as one number has no channel to count, so this cannot be recovered from a record that did not keep one.
  flakiness spread           REFUSED  ACCESS_INSUFFICIENT
                             needs TASK: QUERY|REPLICATE; you have GRADER: RECORD, RECORD: RECORD
                             Remedy: supply task at QUERY; task at REPLICATE. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.
  exploit-family accounting  You have found 1 exploit family in 122 rollout-relation trials-logged finds from verl gsm8k compute_score.  [exploratory]
                             Chao1 puts 0.0% of the families still unseen [nan%, nan%] (0.00 unseen against 1 seen), and that is a floor rather than an estimate.
                             Good-Turing puts the probability that the next find is novel at 0.000 [0.000, 0.000] at 95%, by Esty's variance.
                             f1 = 0, f2 = 0, n = 122
                             (+4 more lines in this instrument's own report)
  variance components        REFUSED  ACCESS_INSUFFICIENT
                             needs GRADER: REPLICATE; you have GRADER: RECORD, RECORD: RECORD
                             Remedy: supply grader at REPLICATE. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.
  effective group size       REFUSED  RECORD_INCOMPLETE
                             the record this card was built from carries no per-group score vectors
                             Remedy: pass the per-group score vectors the run actually produced: `CardInputs(group_scores=GroupScores.of([...]))`. The effective size is the nominal group size discounted by how much of the within-group spread is grader noise, so it is a property of recorded groups and not of the grader alone.
  curl mass                  REFUSED  ACCESS_INSUFFICIENT
                             needs GRADER: QUERY; you have GRADER: RECORD, RECORD: RECORD
                             Remedy: supply grader at QUERY. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.
  Afriat index               REFUSED  SUBSTRATE_MISMATCH
                             this instrument applies to NEURAL_GEN, PROCEDURAL; the grader is PROGRAM
                             Remedy: use an instrument declared for PROGRAM. A PROGRAM grader is a different kind of object, not a harder case of the same one.
  abstention channel         REFUSED  RECORD_INCOMPLETE
                             the record this card was built from carries no recorded score trees
                             Remedy: instrument the grader with `reward_lens.tap` so the per-leaf scores and the abstention channel are recorded, then pass `score_trees=` or `record_groups=` from the recorded groups. A composed score written down as one number has no channel to count, so this cannot be recovered from a record that did not keep one.
