DensityScoreResult#

class cultivars.forecast.scoring.DensityScoreResult(*, names, crps, log_score, energy, squared_error, absolute_error, n_draws)[source]

Bases: _SummaryMixin

One forecast origin’s scores, per horizon and per series.

Every score is negatively oriented – smaller is better – and nothing is averaged: aggregation across horizons or series is the caller’s explicit act, made in daylight. For a predictive sample \(x^{(1)}, \dots, x^{(S)}\) and an outcome \(y\), the scores per cell are the continuous ranked probability score

\[\mathrm{CRPS} = \frac{1}{S} \sum_{s} |x^{(s)} - y| - \frac{1}{2 S^2} \sum_{s, s'} |x^{(s)} - x^{(s')}|,\]

the negative log predictive density \(-\log \hat f(y)\) with \(\hat f\) a Gaussian kernel estimate on the draws, and the squared and absolute errors of the predictive mean and median; per horizon, the energy score

\[\mathrm{ES} = \frac{1}{S} \sum_{s} \|x^{(s)} - y\| - \frac{1}{2 S^2} \sum_{s, s'} \|x^{(s)} - x^{(s')}\|\]

in the Euclidean norm over all \(k\) series at once, the multivariate CRPS. The CRPS and energy score are strictly proper and exact functionals of the sample; the log score is strictly proper but estimated, and is the one most sensitive to tail misfit.

Variables:

  • names – One label per series.

  • crps – (h, k) continuous ranked probability scores, exact from the predictive sample, in each variable’s own units.

  • log_score – (h, k) negative log predictive density at the outcome, the density estimated by a Gaussian kernel on the draws with Silverman’s bandwidth – the module’s one estimated quantity, and the score most sensitive to tail misfit.

  • energy – (h,) energy scores of the joint predictive, the multivariate CRPS.

  • squared_error – (h, k) squared errors of the predictive mean.

  • absolute_error – (h, k) absolute errors of the predictive median.

  • n_draws – Predictive draws scored against.

Note

The point losses ride along because backtests report them, not because they suffice: the squared error scores the mean and the absolute error the median, each the point forecast that is optimal under its loss, and both discard the spread that the proper scores grade. A score is a statement about one realized outcome; a ranking of forecasters needs the scores of many origins, which BacktestResult collects, and none of it is a posterior probability of model truth.

See also

  • DensityScore – the producer.

  • pinball_loss() – the proper score for a quantile forecast, which this record does not carry.

  • losses() – the same CRPS and log score, one value per origin over a backtest.

References

Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359-378.

Gneiting, T., Balabdaoui, F., & Raftery, A. E. (2007). Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society: Series B, 69(2), 243-268.

Example

Standard-normal draws scored against the outcome zero recover the closed forms \(\mathrm{CRPS} = 2\phi(0) - 1/\sqrt{\pi} \approx 0.234\) and \(-\log \phi(0) \approx 0.919\):

>>> import numpy as np
>>> rng = np.random.default_rng(0)
>>> exact = DensityScore(rng.standard_normal((20000, 1)), np.zeros(1)).compute()
>>> exact.n_horizons, exact.n_draws, exact.crps.shape
(1, 20000, (1, 1))
>>> bool(abs(exact.crps[0, 0] - 0.234) < 0.01)
True
>>> bool(abs(exact.log_score[0, 0] - 0.919) < 0.1)
True

A forecaster whose densities are too narrow is penalized by every proper score, the log score most of all:

>>> outcome = rng.standard_normal((4, 2))
>>> honest = DensityScore(rng.standard_normal((2000, 4, 2)), outcome).compute()
>>> narrow = DensityScore(0.3 * rng.standard_normal((2000, 4, 2)), outcome).compute()
>>> honest.crps.shape, honest.energy.shape
((4, 2), (4,))
>>> bool(honest.crps.mean() < narrow.crps.mean())
True
>>> bool(narrow.log_score.mean() / honest.log_score.mean() > 3)
True

Private methods#

_summary_table()

Build the structured summary: one row per series, averaged over horizons.
On this page