Backtest#

class cultivars.forecast.backtest.Backtest(endog, fit, *, horizons=1, start, scheme='expanding', window=None, step=1, predict=None, names=None)[source]#

Run a rolling- or expanding-origin forecasting exercise.

The harness behind every out-of-sample evaluation in the package. Given the full sample and a fitting function from a sample window to a fitted result, it visits the origins

\[o_t = \texttt{start} + t \cdot \texttt{step}, \qquad t = 0, \dots, T - 1, \qquad T = \Bigl\lfloor \frac{n - H - \texttt{start}}{\texttt{step}} \Bigr\rfloor + 1,\]

and at each one estimates on the observations available – all \(o_t\) of them under the expanding scheme, the last \(\texttt{window}\) under the rolling one – forecasts horizons \(1, \dots, H\), and stores the forecast beside the outcome it was for, \(y_{o_t + h - 1}\). The record that comes back, BacktestResult, is therefore aligned by construction, which is the one property the comparison, calibration, and confidence-set tools depend on and cannot verify for themselves.

How the fitted result is asked to forecast is a convention the harness applies for the package’s own families, and a predict hook overrides it for any other. A result exposing forecast_paths(steps, seed=...) (every sampled result with a closed system) is treated as a density forecaster and its draws are kept; one exposing forecast(steps) as a point forecaster. The fitting function receives exactly what the model constructor would: an (m, k) window for a multivariate sample, an (m,) one for a single series.

Parameters:

  • endog (ArrayLike) – The full sample, (n,) or (n, k).

  • fit (Callable[[NDArray[double]], object]) – A function from a sample window (m, k) ((m,) for a single series) to a fitted result, typically lambda w: VAR(w, order=2).fit().

  • horizons (int) – Longest horizon \(H\); every horizon 1 .. horizons is forecast and recorded.

  • start (int) – Observations in the first estimation window. The first origin forecasts rows start .. start + horizons - 1.

  • scheme (str) – "expanding" grows the window from start; "rolling" keeps it at window observations.

  • window (int | None) – Rolling window length; defaults to start. Ignored under the expanding scheme.

  • step (int) – Origins between successive evaluations, at least one.

  • predict (Callable[[object, int, int | None], ArrayLike] | None) – (result, steps, seed) -> forecast for a forecaster the convention does not cover: (n_draws, steps, k) draws for a density forecaster, (steps, k) for a point one. By default forecast_paths(steps, seed=seed) is used when the result offers it, else forecast(steps).

  • names (Sequence[str] | None) – Series labels; default y1 .. yk (y for one series).

Variables:

  • _endog – The validated sample as (n, k) float64, a single series stored as one column.

  • _fit – The fitting function, called once per origin.

  • _horizons – \(H\).

  • _start – Observations in the first estimation window.

  • _scheme – "expanding" or "rolling".

  • _window – The resolved rolling window, start when none was given; recorded on the result under either scheme.

  • _step – Origins between successive evaluations.

  • _predict – The override forecasting hook, or None for the convention.

  • _names – The resolved series labels.

  • _univariate – Whether the sample arrived one-dimensional, so that each window is handed to fit as (m,).

Raises:

  • SpecificationError – If the window, horizon, or step leaves no origin to evaluate, or the scheme is unknown.

  • DimensionError – If the sample is not one- or two-dimensional, or the labels do not match its columns.

  • NumericalError – If the sample is not finite.

Note

Estimation is repeated at every origin; there is no stale-parameter schedule that re-estimates every \(r\) origins and forecasts from old parameters in between, because that needs a result able to forecast from data it was not fitted to, a hook the result family does not carry. step spaces the origins instead, which cuts the cost the same way without a forecast ever being made from a model that has not seen the sample up to its origin. Nothing is estimated until run() is called; constructing the harness only validates the schedule.

Warning

A forecast made with seed=None draws fresh predictive shocks on every run, so two density backtests of the same sample will differ at display precision unless run() is given a seed. Point backtests are deterministic regardless.

See also

References

Tashman, L. J. (2000). Out-of-sample tests of forecasting accuracy: An analysis and review. International Journal of Forecasting, 16(4), 437-450.

West, K. D. (2006). Forecast evaluation. In G. Elliott, C. W. J. Granger, & A. Timmermann (Eds.), Handbook of Economic Forecasting (Vol. 1, pp. 99-134). Elsevier.

Example

A VAR backtested from 100 observations, then a rolling schedule that evaluates every tenth origin on a 60-observation window:

>>> import numpy as np
>>> from cultivars.multivariate.reduced_form.vector_autoregression import VAR
>>> y = np.random.default_rng(0).standard_normal((130, 2))
>>> harness = Backtest(y, lambda w: VAR(w, order=1).fit(), horizons=2, start=100)
>>> harness.origins[:3], harness.origins.shape
(array([100, 101, 102]), (29,))
>>> harness.run().point.shape
(29, 2, 2)
>>> rolling = Backtest(
...     y, lambda w: VAR(w, order=1).fit(), start=100, scheme="rolling", window=60, step=10
... )
>>> rolling.origins, rolling.run().window
(array([100, 110, 120]), 60)

A forecaster from outside the package is backtested through predict; here the historical mean, whose “result” is the window’s mean and whose forecast repeats it at every horizon:

>>> benchmark = Backtest(
...     y[:, 0],
...     lambda w: w.mean(),
...     horizons=2,
...     start=100,
...     predict=lambda mean, steps, seed: np.full(steps, mean),
... )
>>> record = benchmark.run()
>>> record.names, record.point.shape
(('y',), (29, 2, 1))
>>> bool(np.isclose(record.point[0, 0, 0], y[:100, 0].mean()))
True

Attributes#

origins

Observations available at each origin the schedule will evaluate.

Methods#

run(*[, seed])

Estimate at every origin, forecast, and record.
On this page