# hendrycks/math is_equiv
# kind      grader
# source    https://raw.githubusercontent.com/hendrycks/math/main/modeling/math_equivalence.py (sha256 c4101b1f51a2bb65)
# entry     is_equiv
# corpus    MATH-500 gold answers, 240 rollouts
# note      The facet is which way round the two strings are passed. `is_equiv(str1, str2)` normalises both arguments and compares them, so nothing in its shape guarantees the verdict is symmetric; a marker's is. On these 240 pairs it turned out to be, which is why the variance-components field reports no facet variance and not why it was asked.
#
GRADER CARD  hendrycks/math is_equiv
  what a buyer gets before spending money on a grader, and what an auditor gets after

  subject        verifier:95be1ad8dbf1474a1a7d56058e5f2b25
  substrate      PROGRAM    phase  PRE_RUN
  access         GRADER: RECORD, RECORD: RECORD
                 short of D7's stated minimum (GRADER: QUERY|REPLICATE) by GRADER: QUERY|REPLICATE
  trust          exploratory, computed by the gates
  envelope       STATIONARY_GRADER was not measured, so this check did not run. A card assembled across a grader edit describes two programs rather than one. D10 (`env.replay_fidelity`) is what measures whether it was one; pass its ConditionReading as `regime_reading` to have this checked.
  cost           no field priced itself
  not checked    envelope (regime not measured) on the card itself; limit of detection on the card itself

  0 of 13 fields read and 13 refused. Nothing read, so this card is a list of what it would take.

  coverage                   REFUSED  ACCESS_INSUFFICIENT
                             needs GRADER: SOURCE; you have GRADER: RECORD, RECORD: RECORD
                             Remedy: supply grader at SOURCE. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.
  surviving mutants          REFUSED  ACCESS_INSUFFICIENT
                             needs GRADER: SOURCE; you have GRADER: RECORD, RECORD: RECORD
                             Remedy: supply grader at SOURCE. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.
  metamorphic violations     REFUSED  ACCESS_INSUFFICIENT
                             needs GRADER: QUERY; you have GRADER: RECORD, RECORD: RECORD
                             Remedy: supply grader at QUERY. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.
  sensitivity profile        REFUSED  ACCESS_INSUFFICIENT
                             needs GRADER: QUERY; you have GRADER: RECORD, RECORD: RECORD
                             Remedy: supply grader at QUERY. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.
  false-positive catalogue   REFUSED  ACCESS_INSUFFICIENT
                             needs GOLD: QUERY, GRADER: QUERY; you have GRADER: RECORD, RECORD: RECORD
                             Remedy: supply gold at QUERY; grader at QUERY. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.
  silent-zero rate           REFUSED  RECORD_INCOMPLETE
                             the record this card was built from carries no recorded score trees
                             Remedy: instrument the grader with `reward_lens.tap` so the per-leaf scores and the abstention channel are recorded, then pass `score_trees=` or `record_groups=` from the recorded groups. A composed score written down as one number has no channel to count, so this cannot be recovered from a record that did not keep one.
  flakiness spread           REFUSED  ACCESS_INSUFFICIENT
                             needs TASK: QUERY|REPLICATE; you have GRADER: RECORD, RECORD: RECORD
                             Remedy: supply task at QUERY; task at REPLICATE. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.
  exploit-family accounting  REFUSED  ABOVE_LOD_BELOW_LOQ
                             f2 = 0: no family in this log of 259 finds appears exactly twice, so f1**2/(2*f2) is undefined. f1 = 1 over 4 families found.
                             Bound: 0.0
                             Remedy: collect about 259 more finds under the same search and re-run. The doubleton class is what Chao1 divides by, and an equiprobable-family model says a log this length should populate it after roughly that much more searching. The bias-corrected bound f1*(f1-1)/2 = 0.0 unseen families is attached and is a floor, not an estimate. Good-Turing's novelty probability f1/n = 0.004 needs no doubletons and is available now at rung 0.
  variance components        REFUSED  ACCESS_INSUFFICIENT
                             needs GRADER: REPLICATE; you have GRADER: RECORD, RECORD: RECORD
                             Remedy: supply grader at REPLICATE. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.
  effective group size       REFUSED  RECORD_INCOMPLETE
                             the record this card was built from carries no per-group score vectors
                             Remedy: pass the per-group score vectors the run actually produced: `CardInputs(group_scores=GroupScores.of([...]))`. The effective size is the nominal group size discounted by how much of the within-group spread is grader noise, so it is a property of recorded groups and not of the grader alone.
  curl mass                  REFUSED  ACCESS_INSUFFICIENT
                             needs GRADER: QUERY; you have GRADER: RECORD, RECORD: RECORD
                             Remedy: supply grader at QUERY. If you cannot, ask for the same quantity at a lower rung: `reward-lens capabilities` prints which rungs your access reaches and what each costs.
  Afriat index               REFUSED  SUBSTRATE_MISMATCH
                             this instrument applies to NEURAL_GEN, PROCEDURAL; the grader is PROGRAM
                             Remedy: use an instrument declared for PROGRAM. A PROGRAM grader is a different kind of object, not a harder case of the same one.
  abstention channel         REFUSED  RECORD_INCOMPLETE
                             the record this card was built from carries no recorded score trees
                             Remedy: instrument the grader with `reward_lens.tap` so the per-leaf scores and the abstention channel are recorded, then pass `score_trees=` or `record_groups=` from the recorded groups. A composed score written down as one number has no channel to count, so this cannot be recovered from a record that did not keep one.
