Calibration#

class cultivars.forecast.calibration.Calibration(paths, realized, *, names=None)[source]#

Read a rolling forecast record’s calibration.

Takes the stack a driver loop produces – one predictive sample per evaluation origin beside the outcome that origin was forecasting – and asks whether reality landed where the densities said it would. Each outcome is passed through its own predictive distribution,

\[u_t = \hat F_t(y_t),\]

and under correct calibration the \(u_t\) are independent uniforms. compute() records that series and, on its normal quantiles \(z_t = \Phi^{-1}(u_t)\), runs Berkowitz’s (2001) likelihood-ratio test of zero mean, zero slope, and unit variance against a free Gaussian AR(1), one verdict per series.

Parameters:

  • paths (ArrayLike) – (T, n_draws, k) predictive samples, one block per evaluation origin; (T, n_draws) is promoted to k = 1.

  • realized (ArrayLike) – (T, k) outcomes, aligned origin by origin; (T,) is promoted to k = 1.

  • names (tuple[str, ...] | None) – One label per series. Defaults to y1 ... yk.

Variables:

  • _paths – The validated (T, n_draws, k) predictive stack.

  • _realized – The validated (T, k) outcomes.

  • _names – The resolved series labels.

Raises:

  • DimensionError – If the stacks cannot be aligned, or the labels do not match the series.

  • SpecificationError – If the record is too short for the Berkowitz null to mean anything.

  • NumericalError – If any input is not finite.

Note

The input grammar differs from scoring’s on purpose: calibration is a statement across many origins, so the class takes the rolling stack rather than one predictive sample, and refuses fewer than ten origins, below which the three-parameter Berkowitz likelihood has nothing to estimate from. The alignment of paths[t] with realized[t] is the caller’s promise; a BacktestResult keeps it by construction and hands over one horizon’s pair through record().

Warning

The Berkowitz null assumes one-step transforms. A multi-step horizon’s PIT series is serially correlated by construction, which the test reads as miscalibration; read its verdicts as conservative or restrict the formal test to horizon one.

See also

References

Berkowitz, J. (2001). Testing density forecasts, with applications to risk management. Journal of Business & Economic Statistics, 19(4), 465-474.

Diebold, F. X., Gunther, T. A., & Tay, A. S. (1998). Evaluating density forecasts with applications to financial risk management. International Economic Review, 39(4), 863-883.

Example

A synthetic calibrated stack, then the one-step slice of a BVAR backtest fed straight from its record:

>>> import numpy as np
>>> rng = np.random.default_rng(0)
>>> paths = rng.standard_normal((80, 500, 2))
>>> outcomes = rng.standard_normal((80, 2))
>>> record = Calibration(paths, outcomes).compute()
>>> record.pit.shape
(80, 2)
>>> bool(record.berkowitz("y1").pvalue > 0.01)
True
>>> from cultivars.forecast.backtest import Backtest
>>> from cultivars.multivariate.large_dim.bayesian import BVAR
>>> y = np.random.default_rng(0).standard_normal((160, 2))
>>> fit = lambda w: BVAR(w, order=1).fit(n_draws=100, seed=0)
>>> backtest = Backtest(y, fit, horizons=2, start=120, step=2).run(seed=0)
>>> Calibration(*backtest.record(1)).compute().pit.shape
(20, 2)

Methods#

compute()

Transform every origin and test every series.
On this page