How the machinery keeps us honest

1 · The trust ladder and the three gates

Every measurement in reward-lens returns an Evidence object, and every Evidence object carries a trust level: exploratory, calibrated, registered, or adjudicated. The rungs are earned. A bare measurement is exploratory. It becomes calibrated when the instrument that produced it holds a scorecard from runs against planted ground truth. It becomes registered when the prediction it tests was frozen before the evidence existed. It becomes adjudicated when machinery compared it to its frozen threshold and wrote the verdict into the store.

Three gates enforce this. The first computes the trust level from the calibration and registration facts. The second refuses to compare coordinate-dependent quantities across models unless a shared frame is supplied. The third is the study runner: freeze, run, adjudicate, in that order, with the spec hashes checked at adjudication time.

Trust levels are computed by the gates and cannot be set by callers.

2 · The freeze

The freeze manifest records the timestamp, the git sha, the tag, and one hash per spec. The +dirty suffix that appears on the sha in the frozen specs is disclosed in the sha string itself rather than cleaned away.

{
 "created_at": "2026-07-18T23:46:57.951556+00:00",
 "git_sha": "f93f4b5578c221d3d2665f9b305f3e86759be862",
 "tag": "campaign-freeze-v1",
 "code_tree_hash": "code:45d164ec7f6effe9c615daa87c0b13bd",
 "spec_hashes": [
  "spec:c8ac36e72adcc96a3d65b802efbce423",
  "spec:39a99a5ab61d6f8e744a40b3804d21ab",
  "spec:59c457674b4b4f6fc20df702d3e34785",
  "spec:7077cf849ed8492fc327d970aa2ede55",
  "… 23 more"
 ]
}

{
 "frozen_at": "2026-07-18T23:46:57.951556+00:00",
 "git_sha": "f93f4b5578c221d3d2665f9b305f3e86759be862+dirty",
 "spec": {
  "analysis": "campaign.chidrift.analyze",
  "hypotheses": [
   {
    "id": "H-chi-bon",
    "prediction": {
     "ci_excludes": 0,
     "comparator": ">",
     "effect": null,
     "metric": "chi_bon_spearman",
     "threshold": 0.3
    },
    "scoreboard_row": "T9",
    "statement": "chi predicts realized best-of-n feature-drift ordering on a real bank"
   },
   {
    "id": "H-chi-beats-var",
    "prediction": {
     "ci_excludes": 0,
     "comparator": ">",
     "effect": null,
     "metric": "chi_minus_variance_baseline_spearman",
     "threshold": 0
    },
    "scoreboard_row": null,
    "statement": "chi beats a base-variance-only baseline"
   }
  ],
  "id": "campaign-chi-drift",
  "kill_criteria": [
   {
    "comparator": "<",
    "id": "K-chi",
    "metric": "chi_bon_spearman",
    "threshold": 0.3
   }
  ],
  "science": "S03-thermo",
  "subjects": {
   "datasets": [
    "ultrafeedback-bank"
   ],
   "extra": {
    "slice_hashes": {
     "ultrafeedback-bank": "ds:1cb97d4beca432de1793f56a4183a4aa"
    }
   },
   "organisms": [],
   "signals": [
    "skywork-v2-qwen3-8b",
    "skywork-v2-llama31-8b"
   ]
  },
  "title": "Susceptibility chi predicts best-of-n feature drift on a real bank",
  "version": 1
 },
 "spec_hash": "spec:1c12042d2eeb93b23ff9cf27b6433125",
 "study_id": "study:campaign-chi-drift@v1#1c12042d"
}

3 · Adjudication

An adjudication record holds the frozen thresholds and comparators, the observed metrics, and one outcome per hypothesis. A kill criterion is a pre-registered condition under which the card's claim is declared dead; when one fires, the record says so, and the result stands as a registered negative, not a discarded run.

{
 "id": "ev:c978e3958769ecf93dcf079771a3aedb",
 "observable": "campaign.adjudication.CHI-DRIFT",
 "observable_version": "1.0",
 "subject": {
  "signals": [],
  "dataset": null,
  "readout": null,
  "frame": null,
  "interventions": [],
  "extra": {
   "card": "CHI-DRIFT",
   "spec": "campaign-chi-drift"
  }
 },
 "value": {
  "__map__": {
   "card": "CHI-DRIFT",
   "status": "adjudicated",
   "reason": "",
   "outcomes": {
    "__map__": {
     "H-chi-bon": "confirmed",
     "H-chi-beats-var": "confirmed"
    }
   },
   "metrics": {
    "__map__": {
     "chi_bon_spearman": 0.9642857142857145,
     "chi_bon_spearman_ci_low": 0.7500000000000002,
     "chi_bon_spearman_ci_high": 1,
     "chi_bon_spearman_second_model": 1,
     "chi_minus_variance_baseline_spearman": 0.5714285714285714,
     "chi_minus_variance_baseline_spearman_ci_low": 0.07142857142857151,
     "chi_minus_variance_baseline_spearman_ci_high": 0.5714285714285716
    }
   },
   "thresholds": {
    "__map__": {
     "H-chi-bon": 0.3,
     "H-chi-beats-var": 0
    }
   },
   "comparators": {
    "__map__": {
     "H-chi-bon": ">",
     "H-chi-beats-var": ">"
    }
   },
   "kill_thresholds": {
    "__map__": {
     "K-chi": 0.3
    }
   },
   "kill_comparators": {
    "__map__": {
     "K-chi": "<"
    }
   },
   "killed": false,
   "killed_by": {
    "__seq__": []
   },
   "overlay_notes": {
    "__seq__": []
   },
   "staged_violation": false,
   "pre_freeze_violation": false
  }
 },
 "uncertainty": {
  "ci_low": null,
  "ci_high": null,
  "ci_level": null,
  "n": null,
  "n_effective": null,
  "seed_spread": null,
  "method": "none"
 },
 "gauge": "invariant",
 "calibration": null,
 "trust": 2,
 "provenance": {
  "git_sha": "f93f4b5578c221d3d2665f9b305f3e86759be862+dirty",
  "config_hash": null,
  "seeds": [],
  "cost": {
   "gpu_seconds": 0,
   "tokens": 0,
   "wall_seconds": 0
  },
  "oracle_calls": [],
  "parents": [
   "ev:f0436051154b9635b33b4286d25ca559"
  ],
  "study": "study:campaign-chi-drift@v1#1c12042d",
  "extra": {
   "gpu": "cpu",
   "arc": "reporting"
  }
 },
 "created_at": "2026-07-19T18:16:45.257405+00:00"
}

4 · The honest engineering section

Eight inconclusive cards, with their verbatim reasons

• STYLE-RMB: missing intermediate: no intermediate 'campaign.bias.battery' for roster_key='armorm' slice='diagnostic-v3-degradation'; the arc that produces it has not run or its shard was not merged
• PPE-BON: missing intermediate: no intermediate 'campaign.scores' for roster_key='skywork-v2-qwen3-8b' slice='ppe-best-of-k::gpqa'; the arc that produces it has not run or its shard was not merged
• HUMP: missing intermediate: no intermediate 'campaign.recorder.drift' for roster_key='policy-qwen25-1.5b' slice='hump-prompts'; the arc that produces it has not run or its shard was not merged
• GAUGE-E19: resolve_subjects failed: PermissionError(13, 'Permission denied')
• GAUGE-XFAM: resolve_subjects failed: PermissionError(13, 'Permission denied')
• HACK-FORE: missing intermediate: no intermediate 'campaign.index.table' for roster_key='armorm' slice='diagnostic-v3'; the arc that produces it has not run or its shard was not merged
• VALUES-CONTEST: missing intermediate: no intermediate 'campaign.scores' for roster_key='armorm' slice='helpsteer3-pref-2000'; the arc that produces it has not run or its shard was not merged
• T3-FIELD: missing intermediate: no intermediate 'campaign.subspace.flat' for roster_key='skywork-v2-llama31-8b' slice='rb2-full'; the arc that produces it has not run or its shard was not merged

Two hit a permission error in subject resolution; six were missing an intermediate because an arc never ran or its shard never merged. These are engineering gaps, not findings, and the frozen specs they would be adjudicated against are unchanged.

Eight low-power acceptances, recorded at the freeze

A confirmed headline sitting between its threshold and its recorded MDE is a low-power-band detection, and each affected card page says so. A 28th designed card, T3-27B, was never frozen and appears in no ledger row.

Spend

Metered spend was $17.73 across 16,074.1 H100-seconds (about 4.5 H100-hours), summed from the cost fields of the 1363 records in the merged store. The largest arcs: model:skywork-v2-llama31-8b at 3,862 GPU-seconds and model:skywork-v2-qwen3-8b at 2,910 GPU-seconds . The merge report: 1315 records merged, 0 skipped, 0 missing, 0 failed. The claims checker ran against the store at close: 119 claims checked, 0 failures.

The works ledger

Sixty-five feature rows account for every instrument, adapter, index, and loop the campaign touched: 45 PASS or PASS-DEGRADED, 4 FAIL, 7 EXCUSED, 9 NOT-RUN. Every FAIL is a filed bug or a documented limitation, never a blank. Among the seats that ran without earning an Atlas card: the process RM Qwen2.5-Math-PRM-7B, where the dense extractor failed its answer-key validation and K-verif fired, and the generative judge Skywork-Critic-Llama-3.1-8B, the JUDGE-VBC subject.

5 · Failure classes this machinery exists to prevent

The first version of this library produced confident, wrong numbers in seven documented ways. Each failure class now has a design feature that blocks it and a campaign card that demonstrates the block, with the evidence id of the demonstrating run.

6 · Reproduction

What ships today: the 27 frozen specs (manifest, viewable per card), the store projections this site is built from, the 62 adjudication figures, and this site's own data pipeline with its assertions. The campaign harness itself accompanies a future release of the library.