# SWE-bench grading.py get_eval_tests_report
# kind      grader
# source    /home/suhail-nadaf/final-reward/assay-recon/frameworks/swebench/swebench/harness/grading.py (swebench 4.1.0, sha256 88fc500ebaf692a5)
# entry     get_eval_tests_report
# corpus    SWE-bench Lite gold test lists, 240 rollouts
# note      No metamorphic relation is supplied. Both shipped relation sets rewrite a string input, and this grader has none: it takes two mappings. D3 falls back to the source-reading set, every relation of which is inapplicable here, and the card says so rather than reporting a violation rate of zero.
# note      The gold test lists are truncated to 12 fail-to-pass and 24 pass-to-pass entries per instance so that one mutation run is minutes rather than hours. Truncation cannot change a resolution status, which is a conjunction over the lists.
#
GRADER CARD  SWE-bench grading.py
  what a buyer gets before spending money on a grader, and what an auditor gets after

  subject        verifier:ca48315096e82c960386d4784708a672
  substrate      PROGRAM    phase  PRE_RUN
  access         GRADER: RECORD, RECORD: RECORD
                 short of D7's stated minimum (GRADER: QUERY|REPLICATE) by GRADER: QUERY|REPLICATE
  trust          exploratory, computed by the gates
  envelope       STATIONARY_GRADER was not measured, so this check did not run. A card assembled across a grader edit describes two programs rather than one. D10 (`env.replay_fidelity`) is what measures whether it was one; pass its ConditionReading as `regime_reading` to have this checked.
  cost           no field priced itself
  not checked    envelope (regime not measured) on the card itself; limit of detection on the card itself

  0 of 13 fields read and 13 refused. Nothing read, so this card is a list of what it would take.

  coverage                   REFUSED  ACCESS_INSUFFICIENT
                             needs GRADER: SOURCE; you have GRADER: RECORD, RECORD: RECORD
                             Remedy: supply grader at SOURCE. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.
  surviving mutants          REFUSED  ACCESS_INSUFFICIENT
                             needs GRADER: SOURCE; you have GRADER: RECORD, RECORD: RECORD
                             Remedy: supply grader at SOURCE. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.
  metamorphic violations     REFUSED  ACCESS_INSUFFICIENT
                             needs GRADER: QUERY; you have GRADER: RECORD, RECORD: RECORD
                             Remedy: supply grader at QUERY. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.
  sensitivity profile        REFUSED  ACCESS_INSUFFICIENT
                             needs GRADER: QUERY; you have GRADER: RECORD, RECORD: RECORD
                             Remedy: supply grader at QUERY. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.
  false-positive catalogue   REFUSED  ACCESS_INSUFFICIENT
                             needs GOLD: QUERY, GRADER: QUERY; you have GRADER: RECORD, RECORD: RECORD
                             Remedy: supply gold at QUERY; grader at QUERY. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.
  silent-zero rate           REFUSED  RECORD_INCOMPLETE
                             the record this card was built from carries no recorded score trees
                             Remedy: instrument the grader with `reward_lens.tap` so the per-leaf scores and the abstention channel are recorded, then pass `score_trees=` or `record_groups=` from the recorded groups. A composed score written down as one number has no channel to count, so this cannot be recovered from a record that did not keep one.
  flakiness spread           REFUSED  ACCESS_INSUFFICIENT
                             needs TASK: QUERY|REPLICATE; you have GRADER: RECORD, RECORD: RECORD
                             Remedy: supply task at QUERY; task at REPLICATE. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.
  exploit-family accounting  REFUSED  RECORD_INCOMPLETE
                             the record this card was built from carries no exploit log of finds by family
                             Remedy: log every exploit you have already found, with its family and the effort it took, and pass it: `CardInputs(exploit_log=ExploitLog.of([ExploitFind(family=..., effort=...), ...]))`. The unseen mass is a Good-Turing estimate off the frequency spectrum of what has been found, so a blacklist with no counts on it cannot produce one and the fix is where the blacklist is kept.
  variance components        REFUSED  ACCESS_INSUFFICIENT
                             needs GRADER: REPLICATE; you have GRADER: RECORD, RECORD: RECORD
                             Remedy: supply grader at REPLICATE. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.
  effective group size       REFUSED  RECORD_INCOMPLETE
                             the record this card was built from carries no per-group score vectors
                             Remedy: pass the per-group score vectors the run actually produced: `CardInputs(group_scores=GroupScores.of([...]))`. The effective size is the nominal group size discounted by how much of the within-group spread is grader noise, so it is a property of recorded groups and not of the grader alone.
  curl mass                  REFUSED  ACCESS_INSUFFICIENT
                             needs GRADER: QUERY; you have GRADER: RECORD, RECORD: RECORD
                             Remedy: supply grader at QUERY. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.
  Afriat index               REFUSED  SUBSTRATE_MISMATCH
                             this instrument applies to NEURAL_GEN, PROCEDURAL; the grader is PROGRAM
                             Remedy: use an instrument declared for PROGRAM. A PROGRAM grader is a different kind of object, not a harder case of the same one.
  abstention channel         REFUSED  RECORD_INCOMPLETE
                             the record this card was built from carries no recorded score trees
                             Remedy: instrument the grader with `reward_lens.tap` so the per-leaf scores and the abstention channel are recorded, then pass `score_trees=` or `record_groups=` from the recorded groups. A composed score written down as one number has no channel to count, so this cannot be recovered from a record that did not keep one.
