cultivars.forecast.calibration#
Calibration: does reality land where the densities said it would?
A density forecast can score well and still lie about its own uncertainty – too narrow in the tails, biased in the center – and the probability integral transform is how that shows. Feed each realization through its own predictive distribution,
and a calibrated forecaster produces independent uniforms on \((0, 1)\). The shape of the departure is the diagnosis. A U-shaped PIT histogram means overconfidence (reality keeps landing in the tails the forecaster ruled out), a hump means overdispersion, a slope means bias. Berkowitz’s (2001) test makes the reading formal on the normal-quantile scale \(z_t = \Phi^{-1}(u_t)\), where uniformity plus independence becomes the null
a three-restriction Gaussian likelihood ratio with an exact \(\chi^2_3\) reference distribution.
Two commitments shape the surface. First, the input grammar differs
from scoring’s on purpose: calibration is a statement across many
evaluation origins, so Calibration takes the rolling stack –
one predictive sample per origin, one realization per origin – that a
driver loop produces, and refuses fewer than ten origins rather than
fit a three-parameter likelihood to a handful of points. A
BacktestResult supplies exactly
that stack for one horizon through its
record() method, so
the alignment is never the caller’s arithmetic. Second, one caveat is
stated rather than hidden: the Berkowitz null assumes the transforms
are one-step objects; multi-step forecasts overlap mechanically, and
their PIT series carry serial correlation the test will read as
miscalibration. The record says so in its summary notes, and the
formal verdict is best read at horizon one.
Layout. Calibration is the producer and
CalibrationResult the record, which carries the (T, k) PIT
series, one Berkowitz
LikelihoodRatioTest per
series, and the histogram() that draws the
picture. The numerics live in _core: _pit_from_draws evaluates
the predictive sample’s empirical distribution function at the outcome
with the half-count tie convention and a half-draw clip away from the
endpoints, and _berkowitz_likelihood_ratio fits the AR(1) by
conditional maximum likelihood and returns the statistic, its degrees
of freedom, and the p-value.
References
Berkowitz, J. (2001). Testing density forecasts, with applications to risk management. Journal of Business & Economic Statistics, 19(4), 465-474.
Diebold, F. X., Gunther, T. A., & Tay, A. S. (1998). Evaluating density forecasts with applications to financial risk management. International Economic Review, 39(4), 863-883.
Gneiting, T., Balabdaoui, F., & Raftery, A. E. (2007). Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society: Series B, 69(2), 243-268.
Example
A forecaster whose densities are half as wide as they should be, read first by the histogram and then by the test:
>>> import numpy as np
>>> rng = np.random.default_rng(1)
>>> paths = 0.5 * rng.standard_normal((80, 500))
>>> outcomes = rng.standard_normal(80)
>>> record = Calibration(paths, outcomes).compute()
>>> record.histogram(4).ravel()
array([37., 8., 9., 26.])
>>> bool(record.berkowitz("y1").reject(alpha=0.05))
True
The one-step slice of a BVAR backtest, handed over already aligned:
>>> from cultivars.forecast.backtest import Backtest
>>> from cultivars.multivariate.large_dim.bayesian import BVAR
>>> y = np.random.default_rng(0).standard_normal((160, 2))
>>> fit = lambda w: BVAR(w, order=1).fit(n_draws=100, seed=0)
>>> backtest = Backtest(y, fit, horizons=2, start=120, step=2).run(seed=0)
>>> Calibration(*backtest.record(1)).compute().pit.shape
(20, 2)
Classes#
Read a rolling forecast record's calibration. |
|
The PIT series and its formal reading, per series. |