DensityScore#

class cultivars.forecast.scoring.DensityScore(paths, realized, *, names=None)[source]#

Score one origin’s density forecast against what happened.

Constructs from raw arrays – the forecast_paths output of any Bayesian result in the package, or any rival forecaster’s simulated paths – so the grading is model-agnostic by design. The predictive sample \(\{x^{(s)}_{h}\}_{s=1}^{S}\) at each horizon \(h\) stands in for the predictive distribution, and every score in DensityScoreResult is a functional of that sample and the outcome \(y_h\): the CRPS and the energy score exactly, the log score through a kernel density estimate, the point losses through the sample mean and median.

Parameters:

  • paths (ArrayLike) – (n_draws, h, k) predictive paths; a single-horizon (n_draws, k) block is promoted to h = 1.

  • realized (ArrayLike) – (h, k) outcomes over the same horizons; (k,) is promoted to h = 1.

  • names (tuple[str, ...] | None) – One label per series. Defaults to y1 ... yk.

Variables:

  • _paths – The validated (n_draws, h, k) predictive sample.

  • _realized – The validated (h, k) outcomes.

  • _names – The resolved series labels.

Raises:

Note

One origin at a time. The class scores a single predictive sample against a single outcome path; a ranking of forecasters needs many origins, which is Backtest’s job, and its record computes the same CRPS and log score per origin through losses(). A single series must arrive as a column, (n_draws, 1) or (n_draws, h, 1); a one-dimensional (n_draws,) sample is not promoted, because it cannot be told from a single-horizon (n_draws,) sample of k series with the draws axis missing.

See also

References

Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359-378.

Example

Synthetic draws, then a BVAR’s posterior predictive scored against the four observations it held out:

>>> import numpy as np
>>> rng = np.random.default_rng(0)
>>> paths = rng.standard_normal((2000, 4, 2))
>>> outcome = rng.standard_normal((4, 2))
>>> scores = DensityScore(paths, outcome).compute()
>>> scores.crps.shape
(4, 2)
>>> bool(np.all(scores.crps > 0.0))
True
>>> from cultivars.multivariate.large_dim.bayesian import BVAR
>>> y = np.random.default_rng(0).standard_normal((124, 2))
>>> fit = BVAR(y[:120], order=1, names=("gdp", "cpi")).fit(n_draws=200, seed=0)
>>> held_out = DensityScore(fit.forecast_paths(4, seed=0), y[120:], names=fit.names)
>>> record = held_out.compute()
>>> record.names, record.n_horizons, record.n_draws
(('gdp', 'cpi'), 4, 200)

Methods#

compute()

Evaluate every score.
On this page