BacktestResult#

class cultivars.forecast.backtest.BacktestResult(*, names, horizons, origins, realized, point, paths, scheme, window, step)[source]

Bases: _SummaryMixin

The aligned record of a rolling-origin forecasting exercise.

One row per evaluation origin, one slab per horizon, one column per series: what was forecast, what happened, and, for a density forecaster, the predictive draws behind the forecast. Everything downstream, a density score, a calibration record, a Diebold-Mariano or Clark-West comparison, a model confidence set, reads from these arrays, so the alignment the comparison tests cannot verify is guaranteed here by construction: origin \(t\)’s \(h\)-step forecast and the outcome at origins[t] + h - 1 are stored in the same cell,

\[\texttt{point}[t, h - 1, j] = \hat y_{\,o_t + h - 1 \mid o_t,\, j}, \qquad \texttt{realized}[t, h - 1, j] = y_{\,o_t + h - 1,\, j},\]

with \(o_t\) the number of observations the \(t\)-th estimation saw. Every loss the record computes is a function of those two arrays and, for the density losses, the draws, and is returned one value per origin so that a comparison test receives a series it can difference against another backtest’s.

Variables:

  • names – Series labels.

  • horizons – Longest horizon; every horizon 1 .. horizons is kept.

  • origins – (T,) observations available at each origin, so the first forecast target is row origins[t] of the sample.

  • realized – (T, H, k) outcomes.

  • point – (T, H, k) point forecasts – the predictive mean for a density forecaster.

  • paths – (T, S, H, k) predictive draws, or None for a point forecaster.

  • scheme – "expanding" or "rolling".

  • window – Observations each estimation used under a rolling scheme; the first origin’s count under an expanding one.

  • step – Origins between successive evaluations.

Note

The record is the same type for a point and a density forecaster; the difference is whether paths is None, which is_density reads. Every method that needs draws says so in its error rather than returning a degenerate value, so a point backtest handed to a calibration tool fails at the call that needs the draws, with the remedy named. Losses are negatively oriented throughout: smaller is better, including the log score, which is the negative log predictive density.

See also

References

Diebold, F. X., & Mariano, R. S. (1995). Comparing predictive accuracy. Journal of Business & Economic Statistics, 13(3), 253-263.

Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359-378.

Tashman, L. J. (2000). Out-of-sample tests of forecasting accuracy: An analysis and review. International Journal of Forecasting, 16(4), 437-450.

Example

A point backtest of a VAR, expanding from 100 observations, and the aligned loss series a comparison test would take:

>>> import numpy as np
>>> from cultivars.multivariate.reduced_form.vector_autoregression import VAR
>>> rng = np.random.default_rng(0)
>>> y = np.zeros((160, 2))
>>> for t in range(1, 160):
...     y[t] = 0.5 * y[t - 1] + rng.standard_normal(2)
>>> record = Backtest(y, lambda w: VAR(w, order=1).fit(), horizons=2, start=100).run()
>>> record.n_origins, record.horizons, record.k_endog, record.is_density
(59, 2, 2, False)
>>> record.origins[:3], record.realized.shape, record.paths is None
(array([100, 101, 102]), (59, 2, 2), True)
>>> record.losses("squared", horizon=2, name="y1").shape
(59,)
>>> record.rmse.round(2)
array([[1.06, 1.19],
       [1.21, 1.21]])

A density backtest records draws and unlocks the density losses:

>>> from cultivars.multivariate.large_dim.bayesian import BVAR
>>> fit = lambda w: BVAR(w, order=1).fit(n_draws=100, seed=0)
>>> density = Backtest(y, fit, horizons=2, start=120, step=5).run()
>>> density.n_origins, density.n_draws, density.paths.shape
(8, 100, (8, 100, 2, 2))
>>> density.losses("crps", horizon=1).shape, density.record(2)[0].shape
((8, 2), (8, 100, 2))

Private methods#

_density_losses(kind)

(T, H, k) CRPS or log scores, one origin at a time.

_horizon_index(horizon)

The slab index of a one-based horizon, after checking it exists.

_series_index(name)

The column index of a series label, after checking it exists.

_summary_table()

Build the structured summary: one row per horizon and series.
On this page