BacktestResult#

class cultivars.forecast.backtest.BacktestResult(*, names, horizons, origins, realized, point, paths, scheme, window, step)[source]#

The aligned record of a rolling-origin forecasting exercise.

One row per evaluation origin, one slab per horizon, one column per series: what was forecast, what happened, and, for a density forecaster, the predictive draws behind the forecast. Everything downstream, a density score, a calibration record, a Diebold-Mariano or Clark-West comparison, a model confidence set, reads from these arrays, so the alignment the comparison tests cannot verify is guaranteed here by construction: origin \(t\)’s \(h\)-step forecast and the outcome at origins[t] + h - 1 are stored in the same cell,

\[\texttt{point}[t, h - 1, j] = \hat y_{\,o_t + h - 1 \mid o_t,\, j}, \qquad \texttt{realized}[t, h - 1, j] = y_{\,o_t + h - 1,\, j},\]

with \(o_t\) the number of observations the \(t\)-th estimation saw. Every loss the record computes is a function of those two arrays and, for the density losses, the draws, and is returned one value per origin so that a comparison test receives a series it can difference against another backtest’s.

Variables:

  • names – Series labels.

  • horizons – Longest horizon; every horizon 1 .. horizons is kept.

  • origins – (T,) observations available at each origin, so the first forecast target is row origins[t] of the sample.

  • realized – (T, H, k) outcomes.

  • point – (T, H, k) point forecasts – the predictive mean for a density forecaster.

  • paths – (T, S, H, k) predictive draws, or None for a point forecaster.

  • scheme – "expanding" or "rolling".

  • window – Observations each estimation used under a rolling scheme; the first origin’s count under an expanding one.

  • step – Origins between successive evaluations.

Note

The record is the same type for a point and a density forecaster; the difference is whether paths is None, which is_density reads. Every method that needs draws says so in its error rather than returning a degenerate value, so a point backtest handed to a calibration tool fails at the call that needs the draws, with the remedy named. Losses are negatively oriented throughout: smaller is better, including the log score, which is the negative log predictive density.

See also

References

Diebold, F. X., & Mariano, R. S. (1995). Comparing predictive accuracy. Journal of Business & Economic Statistics, 13(3), 253-263.

Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359-378.

Tashman, L. J. (2000). Out-of-sample tests of forecasting accuracy: An analysis and review. International Journal of Forecasting, 16(4), 437-450.

Example

A point backtest of a VAR, expanding from 100 observations, and the aligned loss series a comparison test would take:

>>> import numpy as np
>>> from cultivars.multivariate.reduced_form.vector_autoregression import VAR
>>> rng = np.random.default_rng(0)
>>> y = np.zeros((160, 2))
>>> for t in range(1, 160):
...     y[t] = 0.5 * y[t - 1] + rng.standard_normal(2)
>>> record = Backtest(y, lambda w: VAR(w, order=1).fit(), horizons=2, start=100).run()
>>> record.n_origins, record.horizons, record.k_endog, record.is_density
(59, 2, 2, False)
>>> record.origins[:3], record.realized.shape, record.paths is None
(array([100, 101, 102]), (59, 2, 2), True)
>>> record.losses("squared", horizon=2, name="y1").shape
(59,)
>>> record.rmse.round(2)
array([[1.06, 1.19],
       [1.21, 1.21]])

A density backtest records draws and unlocks the density losses:

>>> from cultivars.multivariate.large_dim.bayesian import BVAR
>>> fit = lambda w: BVAR(w, order=1).fit(n_draws=100, seed=0)
>>> density = Backtest(y, fit, horizons=2, start=120, step=5).run()
>>> density.n_origins, density.n_draws, density.paths.shape
(8, 100, (8, 100, 2, 2))
>>> density.losses("crps", horizon=1).shape, density.record(2)[0].shape
((8, 2), (8, 100, 2))

Attributes#

names

Series labels, in column order; "y1", "y2", .

horizons

Longest horizon \(H\); every horizon ``1 .

origins

(T,) observations available at each origin, ascending, step apart.

realized

(T, H, k) outcomes, realized[t, h - 1] the sample row origins[t] + h - 1.

point

(T, H, k) point forecasts, aligned with realized cell for cell.

paths

(T, S, H, k) predictive draws, or None for a point forecaster.

scheme

"expanding", each estimation on every row up to its origin, or "rolling".

window

Rows each estimation used under a rolling scheme; the first origin's count when expanding.

step

Origins between successive evaluations; 1 evaluates every origin.

errors

(T, H, k) forecast errors point - realized.

is_density

Whether predictive draws were recorded, which the density losses and record() need.

k_endog

Series forecast, \(k\), the last axis of every array.

mae

(H, k) mean absolute errors, averaged over origins.

mean_crps

(H, k) mean continuous ranked probability scores; needs draws.

mean_log_score

(H, k) mean negative log predictive densities; needs draws.

n_draws

Predictive draws per origin, \(S\); zero for a point forecaster.

n_origins

Evaluation origins, \(T\), the length of every loss series.

rmse

(H, k) root mean squared errors, averaged over origins.

Methods#

losses([kind, horizon, name, tau])

One loss series per origin, aligned for a comparison test.

quantile(tau, *[, horizon])

(T, k) quantile forecasts at level \(\tau\) for one horizon.

record([horizon])

One horizon's (paths, realized) as (T, S, k) and (T, k).

summary()

The estimation summary, renderable as text, HTML, or a dataframe.