reward-lens

Open problems

The strongest experiment this project could run does not exist yet.

It would be a single run where the grader is inspected, the estimator is recorded, the policy's feature changes are followed, a prediction is frozen in advance, and that prediction is scored against a real labelled outcome. All five, on the same run.

Reproduce this

Every bundle on this site currently reads that no independent reproduction has been recorded. Changing one of those lines is worth more than any new feature, and it is the one thing this project cannot do for itself.

  1. Is the measurement real

    Python 3.10 or newer. No GPU, no model, no grader call and no network. Every field on every card re-derives from the artifacts in the bundle using the standard library alone. Pointing the same instrument at a scoring program you already have needs the installed library, and that is the next step rather than this one.

    bash 
    unzip grader.zip -d grader && python grader/reproduce.py

    Output

    subject           access     read  refused  remedies  reasons
is_equiv          full          5        8         8  3
is_equiv          log_only      0       13        13  4
swebench          full          5        8         8  2
swebench          log_only      0       13        13  3
verl_gsm8k        full          6        7         7  2
verl_gsm8k        log_only      1       12        12  3
verl_search_r1    full          6        7         7  2
verl_search_r1    log_only      1       12        12  3

full       52 fields, 22 read, 30 refused
             ABOVE_LOD_BELOW_LOQ      1
             RECORD_INCOMPLETE        21
             SUBSTRATE_MISMATCH       8
log_only   52 fields, 2 read, 50 refused
             ABOVE_LOD_BELOW_LOQ      1
             ACCESS_INSUFFICIENT      32
             RECORD_INCOMPLETE        13
             SUBSTRATE_MISMATCH       4

structured refusal records  30
refusals parsed at full access  30
the two agree               yes

    No independent reproduction recorded.

  2. Inside the reward model

    Python 3.10 or newer. No GPU, no model download and no network. This reproduces the rank correlation from the emitted fixture. Reproducing the patching run itself needs weight access and a GPU, and the bundle does not contain the model.

    bash 
    unzip reward-model.zip -d reward-model && python reward-model/reproduce.py

    Output

    pairs joined                360
components ranked both ways 64
dimensions                  12

dimension              recomputed  published   top attribution  moves
code_correctness       -0.441117  -0.441117   mlp_L31         7.0788%
confidence             -0.381206  -0.381206   mlp_L31         0.0000%
correctness            +0.117822  +0.117822   mlp_L31        34.2073%
factuality             -0.258911  -0.258911   mlp_L31         2.3732%
formatting             +0.076205  +0.076205   mlp_L31       235.2342%
helpfulness            -0.305981  -0.305981   mlp_L31         5.0162%
instruction_following  +0.015099  +0.015099   mlp_L31        18.1439%
math_correctness       +0.152893  +0.152893   mlp_L31        43.1881%
refusal_quality        -0.353836  -0.353836   mlp_L31         0.0000%
safety                 -0.289653  -0.289653   mlp_L31         4.0281%
sycophancy             -0.334428  -0.334428   mlp_L31         0.0000%
verbosity              -0.047506  -0.047506   mlp_L31        18.1041%

largest disagreement with the fixture  0.000000  (tolerance 0.01)
within tolerance                       yes

    No independent reproduction recorded.

  3. Watch the run

    Python 3.10 or newer. No GPU, no model and no network.

    bash 
    unzip run.zip -d run && python run/reproduce.py

    Output

    rollouts in the table       25664
rollouts in run-series.json 25664
distinct step indices       401  (1 to 401)
rollouts per step           64
step count the site states  400

series formatOk    401 points   matches run-series.json: true
series hackRate    401 points   matches run-series.json: true
series realized    401 points   matches run-series.json: true
series written     401 points   matches run-series.json: true

    No independent reproduction recorded.

What is unsolved

Each of these states what the question is, what already exists, what would count as an answer, and what the first step is. An entry that cannot say what would count as an answer is not ready and is not listed.

Entries marked as derived come from the library's own registry, where a quantity with no estimator is an open target. The rest are written by hand and say so.

  1. Dumb-baseline bank

    what the question is
    The discrimination achieved on a claim's own task by the strongest member of the six-baseline bank: the area under the ROC curve of that baseline's per-item scores against the claim's labels.
    what already exists
    String match, length, TF-IDF, n-gram diversity, a scaffolded black-box prompt, the gradient-norm peak. Every claim ships against all six. The reason is a single published case: a probe reported at AUC 0.998 on a task where a zero-parameter string match scores 100%, whose author's summary is "the probe detects the hack, and the detection is empty." 1
    what would count as an answer
    The floor a claim has to clear. A claim whose own discrimination does not exceed this has not been shown to detect anything a zero-parameter comparator could not.
    what the first step is
    Dumb-baseline bank

  2. Surrogate conditions and the concomitant of best-of-n

    what the question is
    The expected gold score of the response that maximises the proxy among n draws, which is the concomitant of the nth order statistic, computed from the finite-n distribution theory rather than by simulation.
    what already exists
    Falsifiable conditions on the proxy beat a predicted turning point, and best-of-n Goodhart is a concomitant problem nobody has recognised as one. reward/sequence [raw]
    what would count as an answer
    Best-of-n Goodhart is a concomitant problem, and the exact finite-n answer has been available for forty years. The reading is exact at the stated n rather than asymptotic.
    what the first step is
    Surrogate conditions and the concomitant of best-of-n

  3. The reward-versus-gold frontier

    what the question is
    E_lambda[g] as a function of KL(pi_lambda || pi_0) over the exponential tilt family, estimated on n base-policy rollouts scored by both the proxy r and the gold channel g with w_i = exp(lambda r_i). Both axes are ratios of weighted means, so the curve needs no optimisation run to exist.
    what already exists
    The whole frontier is estimable before any optimisation is run, from n base-policy rollouts scored by both channels. reward/sequence [raw]
    what would count as an answer
    Where the curve turns is where further optimisation against the proxy stops buying gold. It is a statement about g, not about anything g stands for, and it is only defined out to the visibility horizon.
    what the first step is
    The reward-versus-gold frontier

  4. Surrogate conditions and the concomitant of best-of-n

    what the question is
    A per-criterion pass, fail or untestable verdict on the four surrogate-endpoint conditions under which optimising the proxy cannot harm the gold. They are Prentice (1989), Freedman's proportion of treatment effect explained (1992), Buyse and Molenberghs individual-level against trial-level association (1998), and VanderWeele's monotonicity conditions excluding the surrogate paradox (2013).
    what already exists
    Falsifiable conditions on the proxy beat a predicted turning point, and best-of-n Goodhart is a concomitant problem nobody has recognised as one. 1
    what would count as an answer
    A failed criterion names the condition the proxy violates, which is a stronger and more actionable statement than a predicted turning point. Criterion 4 usually returns untestable without an intervention, and reporting that honestly is the point rather than a shortfall.
    what the first step is
    Surrogate conditions and the concomitant of best-of-n

  5. Reward tail index

    what the question is
    The extreme value index of the reward's upper tail, reported with the estimator's plot over the threshold choice rather than as a point estimate at one k. Hill is reported because section 3.0.1 names it and because it is what makes us comparable to the one published measurement; Pickands carries the LIGHT_TAILED verdict, because Hill is exactly scale invariant and not location invariant, and its support is positive so it can never return evidence of light-tailedness.
    what already exists
    The precondition the whole tilt layer rests on, measured rather than assumed. 1
    what would count as an answer
    The LIGHT_TAILED precondition the whole tilt layer rests on, because K(lambda) is finite only if the moment generating function exists. Strictly the condition is gamma <= 0, and an exactly Gaussian tail gives a Hill estimate near 0.196 at these thresholds, which is what the cited 0.20 is consistent with. A defensible estimate needs about 1,570 exceedances, which is 31,400 prompts at q = 0.95; below that the instrument refuses rather than reporting a number from 200 prompts.
    what the first step is
    Reward tail index

  6. The visibility horizon

    what the question is
    The largest KL(pi_lambda || pi_0) at which the Kish effective sample size ESS(lambda) = (sum w_i)^2 / sum w_i^2 still clears a stated floor, default 0.05 of n. Reported in nats rather than in lambda because lambda carries the reciprocal of the reward's scale and is not comparable across graders; the instrument reports lambda_max and n/ESS alongside.
    what already exists
    Where a tilt extrapolation goes blind, which nobody states. nats/sequence
    what would count as an answer
    Past this point the tilt estimate is carried by a handful of rollouts and the instrument declines to answer. It is the reciprocal of the best-of-n coverage coefficient C = 1 + chi^2(pi* || pi_ref) = n/ESS, which the coverage literature leaves unmeasured.
    what the first step is
    The visibility horizon

  7. Decision study: optimal allocation

    what the question is
    The four-tuple (n prompts, K rollouts per prompt, s grader draws per rollout, m repeat calls per draw) that minimises a stated error variance subject to a stated cost constraint, together with the error variance it achieves and the error variance the current allocation achieves. The error variance is one of three: the batch relative error (sigma2(p) + sigma2(delta; s, m)) / (n*K), the batch absolute error which adds sigma2(Delta) - sigma2(delta), or the per-advantage error sigma2(delta; s, m) * (1 - 1/K). sigma2(delta) and sigma2(Delta) are the D-study relative and absolute error variances of the crossed design that produced the components. The plan is a decision and not a measurement: it is exact arithmetic on estimated components and inherits their uncertainty.
    what already exists
    Where the next dollar goes. OPEN/OPEN
    what would count as an answer
    Read the ratio, not the tuple. A plan that moves 30% of the spend from rollouts to grader replications and cuts the error variance by 22% is a 22% cut for free; the same plan on a grader whose sigma2(delta) is already small will move nothing, and that is the answer too.
    what the first step is
    Decision study: optimal allocation

  8. Variance components and gauge R&R

    what the question is
    the fraction of cells of a fully crossed measurement design that were actually observed. One means every subject was scored by every rater at every occasion, which is the design the expected-mean-square inversion is derived for. Below one the inversion is being applied to a design it does not describe.
    what already exists
    Facet decomposition of `Var(r)`. fraction
    what would count as an answer
    what DESIGN_CROSSED is measured by. It is a property of a measurement design and not of a training run, so it is not determinable from a bare record and `measure_regime` reports it as unknown rather than as a pass, which is the third state that exists for exactly this case.
    what the first step is
    Variance components and gauge R&R

  9. Effective group size

    what the question is
    For a group of K rollouts scored on one prompt under one grader configuration: the number of noiselessly scored rollouts that would carry the same amount of signal as the K observed scores. It is K times the generalizability coefficient of a single observed score over the declared universe of grader draws, sigma2(tau) / (sigma2(tau) + sigma2(delta)) at one draw of every facet, which is 1 for a noiseless grader and falls toward 0 as measurement error takes over the observed spread.
    what already exists
    How many independent gradient-relevant observations a group of K rollouts actually gives you. count/group
    what would count as an answer
    Compare it to K. n_eff = 4.2 at K = 16 means eleven point eight rollouts' worth of compute bought grader noise rather than an independent gradient-relevant observation.
    what the first step is
    Effective group size

  10. Grader stochasticity profile

    what the question is
    The probability that a pairwise verdict disagrees with its own modal verdict, as a U-statistic over all repeat combinations rather than over a diagonal pairing. The diagonal depends on an ordering the data does not carry unless the caller declares `paired_occasions`, which is why it is not the default.
    what already exists
    The grader as a distribution, not a number. 1
    what would count as an answer
    How often re-running the same comparison changes which side wins, which is the pairwise cost of the spread `grader.score_sigma` reports. Invariant under `reward.affine`: an affine rescaling of the scores cannot change which of two is larger.
    what the first step is
    Grader stochasticity profile

  11. Variance components and gauge R&R

    what the question is
    100 * sigma_GRR / sigma_total, where sigma_total is the standard deviation of a single score and sigma_GRR is the square root of the total variance minus the variance of the object of measurement. Everything that is not the object is gauge: facet main effects, object-by-facet interactions, facet-by-facet interactions and the residual. A ratio of standard deviations, following the automotive convention, and not of variances.
    what already exists
    Facet decomposition of `Var(r)`. percent
    what would count as an answer
    Under 10% the measurement system is acceptable, 10 to 30 is conditional on what the measurement is for, over 30 is not acceptable for process control. At 61% most of what you are ranking on is the measurement rather than the thing measured.
    what the first step is
    Variance components and gauge R&R

  12. Variance components and gauge R&R

    what the question is
    1.41 * sigma_part / sigma_GRR, truncated to an integer: the number of distinct non-overlapping categories the measurement system can sort the population of objects into. The 1.41 is the square root of two, from the width of the distribution of true values relative to the width of the measurement error.
    what already exists
    Facet decomposition of `Var(r)`. count
    what would count as an answer
    Five or more is the threshold for using a gauge to control a process. Two means the gauge can tell good from bad and nothing finer, so it cannot resolve two adjacent models. One means it cannot do that either.
    what the first step is
    Variance components and gauge R&R

  13. Composition tree and counterfactual composition

    what the question is
    The fraction of rollouts whose composed score moved when a per-prompt constant was added below a named node of the composition tree. `measure/composition/composition.py:508` computes it as n_reached / n over the rollouts that are comparable; an abstaining rollout has no advantage in either arm and is counted in neither direction, because folding it in would make an all-abstaining group look like a perfectly cancelling one.
    what already exists
    What the reward is actually made of, and what it would be without a piece. 1
    what would count as an answer
    Zero or one means the constant reached none of the group or all of it, the group mean moved by the same amount, and the `reward.null` invariance is exact. Anything strictly between them is the invariance failing, and a hard override is the thing that makes it happen, because a constant added below an override never reaches the rollouts the override pinned.
    what the first step is
    Composition tree and counterfactual composition

  14. Grader stochasticity profile

    what the question is
    The pooled within-item standard deviation of a grader's repeated scores at fixed input, on the grader's own scale.
    what already exists
    The grader as a distribution, not a number. reward [raw]
    what would count as an answer
    How much of the score is the grader arguing with itself. Covariant under `reward.affine`, scaling by |a|, which is the whole reason this is not one id with the flip rate.
    what the first step is
    Grader stochasticity profile

  15. Variance components and gauge R&R

    what the question is
    The decomposition of the variance of a single grader score into additive components, one per source of variation in a fully crossed design: the object of measurement, each facet of the measurement procedure, every interaction among them, and a residual. In a crossed object-by-rater-by-occasion design with one observation per cell the components are sigma2(p), sigma2(r), sigma2(o), sigma2(pr), sigma2(po), sigma2(ro) and sigma2(pro,e), estimated by inverting the expected mean squares. The three-way interaction and the residual are one term because a design with one observation per cell cannot separate them. Components are truncated at zero and every truncation is recorded, because a negative estimate means the true component is near zero and the design could not resolve it, which is a different statement from an established zero.
    what already exists
    Facet decomposition of `Var(r)`. variance
    what would count as an answer
    Read the shares, not the values: the values are in the square of whatever units the grader emits and are not comparable across graders. A large sigma2(r) share means the score depends on which judge you drew. A large sigma2(pr) share means the judges disagree about which response is better, which is the term a group-relative estimator cannot cancel.
    what the first step is
    Variance components and gauge R&R

  16. Semantic placebo

    what the question is
    The effect of a coherent semantically irrelevant direction, applied at the same norm and the same dose, divided by the effect of the claimed direction on the same outcome.
    what already exists
    A coherent but irrelevant direction, not a random Gaussian one. Mandatory on every steering and ablation claim, because a vampires-versus-werewolves direction suppressed deployment-time hacking to 0.000 exactly as well as the real direction. 1
    what would count as an answer
    1.0 means a direction with no relation to the claim did the intervention's work, so the claim does not describe what happened. Near 0 means the effect is specific to the claimed direction. Values above 1 happen and mean the placebo did more.
    what the first step is
    Semantic placebo

  17. Noise and angle, per component

    what the question is
    Congruity = cos of the angle between M-transpose alpha and B', per reward component, with Distortion = 1 - Congruity travelling beside it.
    what already exists
    Every reward component needs at least two numbers, a noise and an angle, and no RLHF tooling separates them. correlation/component
    what would count as an answer
    Every reward component needs at least two numbers, a noise and an angle, and no RLHF tooling separates them. A noisy but congruent measure wants a lower weight; a precise but distorted measure wants a different measure. Those are different remedies and one number cannot distinguish them.
    what the first step is
    Noise and angle, per component

  18. The equal-compensation table

    what the question is
    The information coefficient varrho(k) = n(k)/t(k) with n(k) = sigma^2(k)/mu'(t(k),k)^2, per component of a composite reward. After dividing each signal by its own sensitivity mu', the commissions must be equal, so the raw weights must satisfy alpha_i proportional to mu'_i.
    what already exists
    Which component of a composite reward the policy is being paid least to work on. ratio/component
    what would count as an answer
    Whether the reward is mis-weighted, and if so which component is being starved: the one with the lowest alpha_i/mu'_i. The phrase "equal compensation principle" is Milgrom and Roberts (1992) and does not appear in Holmstrom and Milgrom (1991). Section 3.5.2 states the principle backwards and the correct direction names the opposite end on a real panel.
    what the first step is
    The equal-compensation table

  19. Optimal component weights

    what the question is
    The weight a reward component should carry in a composite reward given its measured noise: alpha* = (I + r C'' Sigma)^-1 B', and in the diagonal case alpha*_i = B_i / (1 + r C_ii sigma_i^2). Holmstrom and Milgrom (1991) under CARA and Brownian noise, with the ordering C'' before Sigma, since matrix products do not commute and at least one transcription in this project's own corpus has them the other way round.
    what already exists
    Nobody sets a reward component's weight from that component's measured noise, and the formula that does has been sitting in contract theory since 1991. 1/component
    what would count as an answer
    A component's optimal weight falls as its measurement noise rises, and the shrinkage factor is the same object as the regressional-Goodhart factor Var(X)/(Var(X)+Var(Z)), derived by contract theory as what the principal should do and by the Goodhart literature as what happens if you do not. Ships as a design check with its five assumptions printed, not as a law.
    what the first step is
    Optimal component weights

  20. The sorting cutoff

    what the question is
    The value of the information coefficient varrho at which components should be separated into different incentive contracts rather than summed into one.
    what already exists
    Do not put a noisy judge and a crisp unit test in the same weighted sum, with the threshold at which that stops being a slogan. ratio/component
    what would count as an answer
    Do not put a noisy judge and a crisp unit test in the same weighted sum. Split them across separate optimisation stages, separate models, or separate gradient paths. The optimum is provably asymmetric, so symmetric treatment of components whose noise differs at all is suboptimal.
    what the first step is
    The sorting cutoff

  21. Effective group size

    what the question is
    For a group of K rollouts, Kish's shape factor on the group's centred scores, (sum w)^2 / (K * sum w^2), which is 1 when every rollout deviates from the group mean by the same magnitude and falls toward 0 as the group's gradient signal concentrates in fewer and fewer rollouts. It is computed on the observed scores, so it carries grader noise, and it is reported as a property of the group rather than of the grader for that reason.
    what already exists
    How many independent gradient-relevant observations a group of K rollouts actually gives you. ratio/group
    what would count as an answer
    How evenly a group spends its gradient. A value near 1 means every rollout in the group is contributing comparably to the advantage; a low value means a few rollouts carry the update and the rest are close to the mean. Measured across eleven real reward models it runs 0.7346 to 0.7562 with a spread of 0.0216, which is where the per-grader variation in the old effective-group-size reading was coming from.
    what the first step is
    Effective group size

  22. Interlaboratory comparison

    what the question is
    The between-study variance of a random-effects meta-analysis, by DerSimonian-Laird, Paule-Mandel or restricted maximum likelihood, reported with a Q-profile interval.
    what already exists
    HuggingFace against vLLM, eager against compiled, two seeds, two precisions. Report `s_L` and the Birge ratio beside every reading. OPEN/OPEN
    what would count as an answer
    An estimate of zero is not evidence of homogeneity. At small k, Cochran's Q detects heterogeneity equal to the typical within-study variance well under half the time, so the interval carries the information the point estimate does not.
    what the first step is
    Interlaboratory comparison

What would actually help

  1. An independent reproduction of any result here.

    Every bundle on this site currently reads that no independent reproduction has been recorded. Changing one of those lines is worth more than any new feature, and it is the one thing this project cannot do for itself.

  2. A labelled run to test this against.

    The run in the second act is public and it is one run. The false-positive rate across runs is the open question, and it needs runs this project does not have.

  3. A trainer adapter.

    Two of the twelve are built. The other ten are mechanical work against interfaces that already exist, and each one makes the instruments reachable from a stack they cannot currently see.

  4. Adversarial review of a frozen specification, before it runs.

    A prediction is only worth freezing if someone has tried to break it first. After the run there is nothing useful left to say about the design.

How to reach the work

The repository and the issue tracker are below. The unit of collaboration here is a frozen specification rather than a pull request, so the useful first message is usually about a measurement rather than about code.