cultivars.forecast.backtest#
Rolling-origin backtests: the harness that produces aligned forecasts and outcomes.
Every out-of-sample evaluation is the same loop: hold the sample at an origin, estimate on what is available, forecast the next \(H\) periods, record what happened, advance. With \(o_t\) the number of observations the \(t\)-th estimation saw, the loop visits
and stores, for every horizon \(h \le H\) and series \(j\), the
forecast \(\hat y_{\,o_t + h - 1 \mid o_t,\, j}\) in the same cell as
the outcome \(y_{\,o_t + h - 1,\, j}\). Backtest runs that
loop so that the alignment every downstream test depends on but cannot
check is made once, by construction, and the scoring, calibration,
comparison, and model-confidence-set tools all read from the one
record, BacktestResult.
The model is supplied as a fitting function from a sample window to a
fitted result, so anything that can be fitted to a prefix of the data
can be backtested: lambda y: VAR(y, order=2).fit() or lambda y:
BVAR(y, order=4).fit(n_draws=500). A result exposing
forecast_paths(steps, seed=...) is backtested as a density forecaster
and its draws are recorded; one exposing forecast(steps) as a point
forecaster. A predict callable overrides that convention for a
forecaster from outside the package.
Two commitments shape the surface. First, estimation is repeated at
every origin. A stale-parameter schedule – re-estimating every
\(r\) origins and forecasting from old parameters in between –
needs a result that can forecast from data it was not fitted to, a hook
the result family does not carry; the step argument spaces the
origins instead, which cuts cost the same way without a forecast ever
being made from a model that has not seen the sample up to its origin.
Second, a loss is a series, not a number. Every loss the record
computes – squared, absolute, pinball, CRPS, log score – comes back
one value per origin, because a comparison test needs the difference of
two such series and its serial correlation, and a summary statistic
would have thrown away exactly what the test uses; the (H, k)
averages are conveniences on top.
Layout. Backtest is the harness and BacktestResult
the record it returns. The scores live in _core: _crps_from_draws
is the exact sample CRPS, _kernel_log_score the negative log
predictive density by Gaussian kernel, and _pinball_loss the
quantile loss; PredictiveResult is the protocol that marks a result
as a density forecaster, and _validate_names / _variable_names
resolve the series labels. _SummaryMixin
gives the record its summary(), str(), and notebook rendering.
References
Diebold, F. X., & Mariano, R. S. (1995). Comparing predictive accuracy. Journal of Business & Economic Statistics, 13(3), 253-263.
Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359-378.
Tashman, L. J. (2000). Out-of-sample tests of forecasting accuracy: An analysis and review. International Journal of Forecasting, 16(4), 437-450.
West, K. D. (2006). Forecast evaluation. In G. Elliott, C. W. J. Granger, & A. Timmermann (Eds.), Handbook of Economic Forecasting (Vol. 1, pp. 99-134). Elsevier.
Example
A VAR backtested from 100 observations, and the aligned loss series a comparison test would take:
>>> import numpy as np
>>> from cultivars.multivariate.reduced_form.vector_autoregression import VAR
>>> rng = np.random.default_rng(0)
>>> y = np.zeros((160, 2))
>>> for t in range(1, 160):
... y[t] = 0.5 * y[t - 1] + rng.standard_normal(2)
>>> record = Backtest(y, lambda w: VAR(w, order=1).fit(), horizons=2, start=100).run()
>>> record.n_origins, record.realized.shape, record.is_density
(59, (59, 2, 2), False)
>>> record.losses("squared", horizon=1, name="y1").shape
(59,)
The same sample under a density forecaster, which unlocks the proper scoring rules and the calibration record:
>>> from cultivars.multivariate.large_dim.bayesian import BVAR
>>> fit = lambda w: BVAR(w, order=1).fit(n_draws=100, seed=0)
>>> density = Backtest(y, fit, horizons=2, start=120, step=5).run(seed=0)
>>> density.n_origins, density.n_draws, density.mean_crps.shape
(8, 100, (2, 2))
Classes#
Run a rolling- or expanding-origin forecasting exercise. |
|
The aligned record of a rolling-origin forecasting exercise. |