Module calib 

Prequential calibration: is the model’s confidence honest?

Every duel is forecast before it is answered — record_duel scores the posterior’s P(A wins) and only then appends the observation — so these are genuinely out-of-sample, one-step-ahead predictions. What was done with them was not: a running count of p > 0.5 outcomes is accuracy, and accuracy is not a proper scoring rule. A model that says 0.51 every time and is right 51 % of the time scores identically to one that says 0.99 and is right 51 % of the time. Worse, an information-seeking acquisition function deliberately picks pairs near p = 0.5, so the hit rate is pinned near 50 % by construction — a perfectly calibrated model looks like a coin flip, and the user concludes it is not learning.

Two honest replacements:

• Brier score B = mean (p_chosen − 1)², reported as skill 1 − B/0.25 against the always-0.5 baseline. Proper, bounded, and it moves as sharpness improves rather than only as accuracy does.
• A reliability diagram: bin the forecasts and compare predicted with observed frequency. This is the display that makes calibration legible — the diagonal is the claim, the bars are the evidence.

And a selection-bias fix, because the acquisition function chooses which duels get scored: a fraction of duels are drawn uniformly at random and flagged (Forecast::random_check). Calibration on those is unbiased, and it is reported separately. It costs a small share of the query budget and it is the only number here that means what it says without an asterisk.

Structs

Calibration

Calibration summary over a set of forecasts.

Forecast

One out-of-sample duel forecast, recorded before the answer was known.

ProvenanceScore

One provenance’s slice of the forecast stream.

ReliabilityBin

One bucket of the reliability diagram.

Functions

calibration

Summarize a forecast stream.