Backtest.run

Backtest.run(*, seed=None)

Estimate at every origin, forecast, and record.

The loop the class exists for. At each origin the fitting function is called on the window, the result is asked to forecast through _forecast(), and the forecast is stored beside the outcomes endog[origin : origin + horizons]. Whether the exercise is a point or a density backtest is decided by the first origin’s forecast and then enforced: every origin must produce the same kind, and every density origin the same number of draws, so that the record’s arrays are rectangular. The point forecast recorded for a density origin is the mean of its draws.

Parameters:

seed (int | None) – Base seed for the predictive draws; origin t uses seed + t so that repeated runs reproduce and origins do not share shocks. None leaves every origin’s draws unseeded.

Return type:

BacktestResult

Returns:

The BacktestResult.

Raises:

• SpecificationError – If a forecaster cannot be read.

• DimensionError – If a forecast has an unexpected shape, or origins disagree on the number of draws or on whether they produce draws at all.

• NumericalError – If a forecast is not finite.

Example

>>> import numpy as np
>>> from cultivars.multivariate.large_dim.bayesian import BVAR
>>> y = np.random.default_rng(0).standard_normal((130, 2))
>>> fit = lambda w: BVAR(w, order=1).fit(n_draws=20, seed=0)
>>> harness = Backtest(y, fit, horizons=2, start=120, step=5)
>>> first, again = harness.run(seed=1), harness.run(seed=1)
>>> first.paths.shape, bool(np.array_equal(first.paths, again.paths))
((2, 20, 2, 2), True)
>>> bool(np.allclose(first.point, first.paths.mean(axis=1)))
True

A hook whose number of draws changes between origins is refused rather than padded:

>>> draws = lambda result, h, seed: np.zeros((seed + 2, h, 2))
>>> uneven = Backtest(y, lambda w: None, start=100, step=10, predict=draws)
>>> uneven.run(seed=0)
Traceback (most recent call last):
    ...
cultivars.exceptions.DimensionError: origin 110 produced 3 draws; earlier origins ...