# verl reward_score/search_r1_like_qa_em.py compute_score
# kind      environment
# source    /home/suhail-nadaf/final-reward/assay-recon/frameworks/verl/verl/utils/reward_score/search_r1_like_qa_em.py (sha256 8070e4d07719f23e)
# entry     compute_score
# corpus    NQ-Open validation split, 240 rollouts
# note      The facet is the match rule: `compute_score` requires the normalised answer to equal a gold exactly and `compute_score_subem`, in the same module, accepts a gold that is a substring of it. Which one runs is a config choice in the environment.
# note      Standard output is muffled while this card builds. The grader prints on a random draw, D1 calls it once per rollout and D2 once per rollout per mutant, so a build that did not muffle it would emit tens of thousands of lines. The draw itself is counted below, because it comes out of the process-wide stream and that is the part that is not cosmetic.
#
GRADER CARD  verl Search-R1 QA exact match
  what a buyer gets before spending money on a grader, and what an auditor gets after

  subject        verifier:e7a8ab3641ad91c4c5a1a68dde13b0c4
  substrate      PROGRAM    phase  PRE_RUN
  access         GOLD: QUERY, GRADER: RECORD|QUERY|REPLICATE|SOURCE, RECORD: RECORD, TASK: QUERY|REPLICATE|SOURCE
                 meets D7's stated minimum (GRADER: QUERY|REPLICATE)
  trust          exploratory, computed by the gates
  lowest reading exploratory
  envelope       STATIONARY_GRADER was not measured, so this check did not run. A card assembled across a grader edit describes two programs rather than one. D10 (`env.replay_fidelity`) is what measures whether it was one; pass its ConditionReading as `regime_reading` to have this checked.
  cost           2,400 grader calls, which is a floor: 5 of the 6 fields that read do not model their own cost
  not checked    envelope (regime not measured) on 5 field(s) and the card itself; limit of detection on 6 field(s) and the card itself

  6 of 13 fields read and 7 refused. The lowest trust among them is exploratory.

  coverage                   Across 240 rollouts compute_score has never taken 57% of its branches (17 of 30). Those are behaviours it cannot distinguish.  [exploratory]
                             never fired: em_check.clause1, subem_check.clause1, subem_check.clause2, extract_solution.clause1, compute_score.clause3, compute_score.clause5, compute_score_subem.clause1, compute_score_subem.clause2, compute_score_subem.clause3
                             a random sample of the same size reaches 0.433 against the corpus's 0.433
  surviving mutants          Of 60 mutants of compute_score, 34 survive: 34 ways it could be wrong that no rollout in your corpus would reveal.  [exploratory]
                             mutation score 0.433 over 60 competent
                             reproducers withheld. this payload enumerates concrete ways to make the grader wrong. It is written to the store with a sensitive flag, excluded from rendered artifacts unless explicitly requested, and not published without a recorded decision.
  metamorphic violations     metamorphic violations on grade  [exploratory]
                             2 violations over 362 applicable rollout-relation pairs (0.55%)
                             corpus 240 rollouts, 120 accepted at score > 0
                             baseline (identity transformation): 0 violations
                             (+4 more lines in this instrument's own report)
  sensitivity profile        REFUSED  RECORD_INCOMPLETE
                             the record this card was built from carries no rubric scorer over named numeric inputs and no declared input ranges
                             Remedy: expose the grader as a scorer over named numeric inputs and declare their ranges: `CardInputs(scorer=fn, rubric_inputs=[RubricInput('helpfulness', 0.0, 1.0), ...])`. A Sobol decomposition apportions output variance across an input space, so a grader with no numeric input space has no sensitivity profile rather than a flat one.
  false-positive catalogue   REFUSED  RECORD_INCOMPLETE
                             the record this card was built from carries no stricter reference oracle and no seed rollouts to search around
                             Remedy: supply a callable grader, a stricter reference that decides the same question, and seed rollouts to search around: `CardInputs(grader=fn, reference=StrictReference('exact', decide, basis='...'), fp_seeds=[...])`. A false positive is a disagreement with a stricter oracle, so without one there is no quantity here, only a distribution of scores.
  silent-zero rate           REFUSED  RECORD_INCOMPLETE
                             the record this card was built from carries no recorded score trees
                             Remedy: instrument the grader with `reward_lens.tap` so the per-leaf scores and the abstention channel are recorded, then pass `score_trees=` or `record_groups=` from the recorded groups. A composed score written down as one number has no channel to count, so this cannot be recovered from a record that did not keep one.
  flakiness spread           The same policy on the same task scored 50.0% on every one of 3 identical runs across 240 task(s). This environment is deterministic here, so a measured delta is not competing with its spread.  [exploratory]
  exploit-family accounting  You have found 1 exploit family in 2 rollout-relation trials-logged finds from verl search_r1 compute_score.  [exploratory]
                             Chao1 puts 0.0% of the families still unseen [nan%, nan%] (0.00 unseen against 1 seen), and that is a floor rather than an estimate.
                             Good-Turing puts the probability that the next find is novel at 0.000 [0.000, 1.000] at 95%, by Esty's variance.
                             f1 = 0, f2 = 1, n = 2
                             (+4 more lines in this instrument's own report)
  variance components        1% of your score variance is match rule disagreement plus residual. %GRR = 9%, ndc = 15. This gauge resolves at least 15 distinct levels of the thing being scored.  [exploratory]
  effective group size       REFUSED  RECORD_INCOMPLETE
                             the record this card was built from carries no per-group score vectors
                             Remedy: pass the per-group score vectors the run actually produced: `CardInputs(group_scores=GroupScores.of([...]))`. The effective size is the nominal group size discounted by how much of the within-group spread is grader noise, so it is a property of recorded groups and not of the grader alone.
  curl mass                  REFUSED  SUBSTRATE_MISMATCH
                             this instrument applies to NEURAL_GEN, PROCEDURAL; the grader is PROGRAM
                             Remedy: use an instrument declared for PROGRAM. A PROGRAM grader is a different kind of object, not a harder case of the same one.
  Afriat index               REFUSED  SUBSTRATE_MISMATCH
                             this instrument applies to NEURAL_GEN, PROCEDURAL; the grader is PROGRAM
                             Remedy: use an instrument declared for PROGRAM. A PROGRAM grader is a different kind of object, not a harder case of the same one.
  abstention channel         REFUSED  RECORD_INCOMPLETE
                             the record this card was built from carries no recorded score trees
                             Remedy: instrument the grader with `reward_lens.tap` so the per-leaf scores and the abstention channel are recorded, then pass `score_trees=` or `record_groups=` from the recorded groups. A composed score written down as one number has no channel to count, so this cannot be recovered from a record that did not keep one.

  1 field(s) withheld their reproducers: surviving mutants. Pass include_sensitive=True with a recorded DisclosureDecision on the payload to see them.
