# hendrycks/math is_equiv
# kind      grader
# source    https://raw.githubusercontent.com/hendrycks/math/main/modeling/math_equivalence.py (sha256 c4101b1f51a2bb65)
# entry     is_equiv
# corpus    MATH-500 gold answers, 240 rollouts
# note      The facet is which way round the two strings are passed. `is_equiv(str1, str2)` normalises both arguments and compares them, so nothing in its shape guarantees the verdict is symmetric; a marker's is. On these 240 pairs it turned out to be, which is why the variance-components field reports no facet variance and not why it was asked.
#
GRADER CARD  hendrycks/math is_equiv
  what a buyer gets before spending money on a grader, and what an auditor gets after

  subject        verifier:95be1ad8dbf1474a1a7d56058e5f2b25
  substrate      PROGRAM    phase  PRE_RUN
  access         GOLD: QUERY, GRADER: RECORD|QUERY|REPLICATE|SOURCE, RECORD: RECORD, TASK: QUERY|REPLICATE|SOURCE
                 meets D7's stated minimum (GRADER: QUERY|REPLICATE)
  trust          exploratory, computed by the gates
  lowest reading exploratory
  envelope       STATIONARY_GRADER was not measured, so this check did not run. A card assembled across a grader edit describes two programs rather than one. D10 (`env.replay_fidelity`) is what measures whether it was one; pass its ConditionReading as `regime_reading` to have this checked.
  cost           2,400 grader calls, which is a floor: 4 of the 5 fields that read do not model their own cost
  not checked    envelope (regime not measured) on 4 field(s) and the card itself; limit of detection on 5 field(s) and the card itself

  5 of 13 fields read and 8 refused. The lowest trust among them is exploratory.

  coverage                   Across 240 rollouts is_equiv has never taken 32% of its branches (12 of 38). Those are behaviours it cannot distinguish.  [exploratory]
                             never fired: _fix_fracs.clause4, _fix_fracs.clause5, _remove_right_units.clause1, _strip_string.clause1, _strip_string.clause5, is_equiv.clause1, is_equiv.clause2, is_equiv.clause3
                             a random sample of the same size reaches 0.653 against the corpus's 0.684
  surviving mutants          Of 60 mutants of is_equiv, 60 survive: 60 ways it could be wrong that no rollout in your corpus would reveal.  [exploratory]
                             mutation score 0.000 over 60 competent
                             reproducers withheld. this payload enumerates concrete ways to make the grader wrong. It is written to the store with a sensitive flag, excluded from rendered artifacts unless explicitly requested, and not published without a recorded decision.
  metamorphic violations     metamorphic violations on grade  [exploratory]
                             259 violations over 378 applicable rollout-relation pairs (68.52%)
                             corpus 240 rollouts, 120 accepted at score > 0
                             baseline (identity transformation): 0 violations
                             (+4 more lines in this instrument's own report)
  sensitivity profile        REFUSED  RECORD_INCOMPLETE
                             the record this card was built from carries no rubric scorer over named numeric inputs and no declared input ranges
                             Remedy: expose the grader as a scorer over named numeric inputs and declare their ranges: `CardInputs(scorer=fn, rubric_inputs=[RubricInput('helpfulness', 0.0, 1.0), ...])`. A Sobol decomposition apportions output variance across an input space, so a grader with no numeric input space has no sensitivity profile rather than a flat one.
  false-positive catalogue   REFUSED  RECORD_INCOMPLETE
                             the record this card was built from carries no stricter reference oracle and no seed rollouts to search around
                             Remedy: supply a callable grader, a stricter reference that decides the same question, and seed rollouts to search around: `CardInputs(grader=fn, reference=StrictReference('exact', decide, basis='...'), fp_seeds=[...])`. A false positive is a disagreement with a stricter oracle, so without one there is no quantity here, only a distribution of scores.
  silent-zero rate           REFUSED  RECORD_INCOMPLETE
                             the record this card was built from carries no recorded score trees
                             Remedy: instrument the grader with `reward_lens.tap` so the per-leaf scores and the abstention channel are recorded, then pass `score_trees=` or `record_groups=` from the recorded groups. A composed score written down as one number has no channel to count, so this cannot be recovered from a record that did not keep one.
  flakiness spread           The same policy on the same task scored 50.0% on every one of 3 identical runs across 240 task(s). This environment is deterministic here, so a measured delta is not competing with its spread.  [exploratory]
  exploit-family accounting  REFUSED  ABOVE_LOD_BELOW_LOQ
                             f2 = 0: no family in this log of 259 finds appears exactly twice, so f1**2/(2*f2) is undefined. f1 = 1 over 4 families found.
                             Bound: 0.0
                             Remedy: collect about 259 more finds under the same search and re-run. The doubleton class is what Chao1 divides by, and an equiprobable-family model says a log this length should populate it after roughly that much more searching. The bias-corrected bound f1*(f1-1)/2 = 0.0 unseen families is attached and is a floor, not an estimate. Good-Turing's novelty probability f1/n = 0.004 needs no doubletons and is available now at rung 0.
  variance components        0% of your score variance is which argument order you drew. %GRR = 0%, ndc = 2147483647. Every distinct value of the thing being scored is resolvable. This design has one observation per cell, so it carries no replication to estimate repeatability from and a measured zero here is the weakest possible evidence for it. Vary a facet and score again, or read the flakiness spread.  [exploratory]
  effective group size       REFUSED  RECORD_INCOMPLETE
                             the record this card was built from carries no per-group score vectors
                             Remedy: pass the per-group score vectors the run actually produced: `CardInputs(group_scores=GroupScores.of([...]))`. The effective size is the nominal group size discounted by how much of the within-group spread is grader noise, so it is a property of recorded groups and not of the grader alone.
  curl mass                  REFUSED  SUBSTRATE_MISMATCH
                             this instrument applies to NEURAL_GEN, PROCEDURAL; the grader is PROGRAM
                             Remedy: use an instrument declared for PROGRAM. A PROGRAM grader is a different kind of object, not a harder case of the same one.
  Afriat index               REFUSED  SUBSTRATE_MISMATCH
                             this instrument applies to NEURAL_GEN, PROCEDURAL; the grader is PROGRAM
                             Remedy: use an instrument declared for PROGRAM. A PROGRAM grader is a different kind of object, not a harder case of the same one.
  abstention channel         REFUSED  RECORD_INCOMPLETE
                             the record this card was built from carries no recorded score trees
                             Remedy: instrument the grader with `reward_lens.tap` so the per-leaf scores and the abstention channel are recorded, then pass `score_trees=` or `record_groups=` from the recorded groups. A composed score written down as one number has no channel to count, so this cannot be recovered from a record that did not keep one.

  1 field(s) withheld their reproducers: surviving mutants. Pass include_sensitive=True with a recorded DisclosureDecision on the payload to see them.
