Calibration

Every duel is forecast before it is answered. This page is how those forecasts are scored, and why the obvious metric would have lied.

Prequential by construction

record_duel scores the posterior's $P(A \text{ wins})$ and only then appends the observation. So every forecast is an out-of-sample, one-step-ahead prediction: the model has never seen the answer it is being scored on.

Why not accuracy

A running count of $p > 0.5$ outcomes is accuracy, and accuracy is not a proper scoring rule. Two failures, and the second is fatal here:

It cannot see sharpness. A model that says 0.51 every time and is right 51% of the time scores identically to one that says 0.99 and is right 51% of the time. The second is wildly overconfident and accuracy cannot tell you.

It is pinned near 50% by the acquisition rule. An information-seeking rule deliberately picks pairs near $p = 0.5$, because those are the questions worth asking. So the hit rate sits near chance by construction: a perfectly calibrated model looks like a coin flip, and the user concludes it is not learning.

The second point is what makes accuracy harmful rather than merely crude: it penalizes the search for doing its job.

hit_rate is still computed and shown, only so the interface can display how misleading it is next to the real number.

Brier score and skill

$$B = \frac{1}{n}\sum_{i} \big(p_i^{\text{chosen}} - 1\big)^2$$

where $p^{\text{chosen}}$ is the probability the model gave to the option the user actually picked. Lower is better; $B = 0.25$ is what always saying 0.5 scores.

Reported as skill against that baseline:

$$\text{skill} = 1 - \frac{B}{0.25}$$

Brier is proper and bounded, and it moves as sharpness improves rather than only as accuracy does, which is the property accuracy lacked.

Log-loss, and what it may not be compared across

$$\text{LL} = \frac{1}{n}\sum_i -\log p_i^{\text{chosen}} \quad \text{(nats)}$$

Baseline $\log 2 \approx 0.693$.

Comparable across time for one acquisition rule. Not comparable across acquisition rules: an information-seeking rule serves duels near $p = 0.5$, which carry the highest log-loss by construction. Comparing two rules on their own self-chosen question sets would score the willingness to ask hard questions as a failure.

check_log_loss is the version for that comparison. See below.

The selection-bias fix

The acquisition function chooses which duels get scored, which means overall skill is measured on a question set the model helped select. That is circular.

So a fraction of duels are drawn uniformly at random and flagged Forecast::random_check. The app marks them ◇ unbiased probe, and calibration restricted to those is unbiased:

It costs a small share of the query budget and it is the only number here that means what it says unqualified.

The reliability diagram

Five buckets over $P(A \text{ wins})$ (N_BINS = 5, the most a small session can fill without every bucket being noise). Each bucket reports:

Plotted, the diagonal is the claim and the dots are the reality. This is the display that makes calibration legible: a single number cannot distinguish "overconfident at the top end" from "underconfident in the middle", and the shape of the failure is what tells you what to do about it.

The app draws a whisker per bucket for how much a bucket that size could wobble by chance, so a dot off the diagonal with a whisker crossing it is not yet evidence of anything.

By provenance

The same scores, split by how the answer was collected:

pub struct ProvenanceScore {
    pub provenance: String,  // "duel" | "heard_edit" | "self_report"
    pub n: usize,
    pub brier: f64,
    pub log_loss: f64,
    pub skill: f64,
}

The comparison this exists for: a hand edit committed through a heard duel and one committed by ticking my edit is better make the same claim in the log, and there is no reason to believe they are equally reliable. Scoring them against forecasts the model made before either answer arrived is the only way to find out which, and it costs one tag.

Empty streams are omitted, so a session that has never committed a hand edit carries exactly one row.

Interpreting it

The user-facing version of this table is in Reading what it learned.

Why the number can look bad

Committing to a forecast before each answer and then reporting the error against a proper scoring rule means the model can publicly fail, and early on it does.

That is what makes the number worth reading later, and it is why the app shows "not beating a coin flip yet" rather than hiding the metric until it flatters.