CalibrationResult#

class cultivars.forecast.calibration.CalibrationResult(*, names, pit, tests)[source]#

The PIT series and its formal reading, per series.

One probability integral transform per evaluation origin and series,

\[u_{t,j} = F_{t,j}(y_{t,j}) = \frac{1}{S} \sum_{s=1}^{S} \mathbb{1}\{x_{t,j}^{(s)} < y_{t,j}\} + \frac{1}{2S} \sum_{s=1}^{S} \mathbb{1}\{x_{t,j}^{(s)} = y_{t,j}\},\]

the predictive sample’s empirical distribution function at the outcome, and beside the series one Berkowitz verdict per column. If the predictive distributions were correct, the \(u_{t,j}\) are independent uniforms on \((0, 1)\); the departure’s shape is the diagnosis, and histogram() draws it. The verdict is Berkowitz’s likelihood ratio on the normal quantiles \(z_{t} = \Phi^{-1}(u_{t})\), which under the null are independent standard normals: an AR(1) with free mean, slope, and variance is fitted and tested against zero mean, zero slope, and unit variance, a three-restriction \(\chi^2\) test.

Variables:

  • names – One label per series.

  • pit – (T, k) probability integral transforms, one per evaluation origin, interior to (0, 1). Uniform under correct calibration.

  • tests – One Berkowitz likelihood-ratio verdict per series, in names order.

Note

The transforms are clipped half a draw’s probability away from the endpoints, 0.5 / S and 1 - 0.5 / S, so that the normal quantile the test needs is always finite; a finite sample cannot distinguish an outcome beyond every draw from one at the edge. The Berkowitz null assumes one-step transforms. Multi-step forecasts overlap mechanically, and the serial correlation their PIT series inherits reads as miscalibration here; hand the record one horizon at a time and read the multi-step verdicts as conservative.

See also

References

Berkowitz, J. (2001). Testing density forecasts, with applications to risk management. Journal of Business & Economic Statistics, 19(4), 465-474.

Diebold, F. X., Gunther, T. A., & Tay, A. S. (1998). Evaluating density forecasts with applications to financial risk management. International Economic Review, 39(4), 863-883.

Example

A calibrated forecaster, then one whose densities are half as wide as they should be, which the U-shaped histogram and the Berkowitz test both catch:

>>> import numpy as np
>>> rng = np.random.default_rng(0)
>>> paths = rng.standard_normal((80, 500, 2))
>>> outcomes = rng.standard_normal((80, 2))
>>> record = Calibration(paths, outcomes).compute()
>>> record.n_origins, record.pit.shape
(80, (80, 2))
>>> round(float(record.pit[:, 0].mean()), 2), bool(record.berkowitz("y1").pvalue > 0.05)
(0.48, True)
>>> rng = np.random.default_rng(1)
>>> narrow = Calibration(0.5 * rng.standard_normal((80, 500)), rng.standard_normal(80))
>>> overconfident = narrow.compute()
>>> overconfident.histogram(4).ravel()
array([37.,  8.,  9., 26.])
>>> bool(overconfident.berkowitz("y1").pvalue < 0.001)
True

Attributes#

names

Series labels, in column order; "y1", "y2", .

pit

(T, k) probability integral transforms, one row per origin.

tests

One Berkowitz likelihood-ratio record per series, df = 3, in names order.

n_origins

Evaluation origins, \(T\), the length of each PIT series.

Methods#

berkowitz(name)

One series' Berkowitz verdict.

histogram([bins])

PIT histogram counts on equal-width bins over (0, 1).

summary()

The estimation summary, renderable as text, HTML, or a dataframe.