cultivars.forecast#
Forecast evaluation: scoring, comparison, and calibration of densities.
A fitted model’s forecast is a claim, and this package is where claims
are graded. The workflow runs in one direction. A
Backtest re-estimates a model at
every origin of a rolling or expanding schedule and records forecasts
beside outcomes, aligned by construction; from that record, or from raw
arrays a rival forecaster produced, the other modules read. Scoring
grades one origin’s density with strictly proper rules; calibration asks
across many origins whether reality landed where the densities said;
comparison tests whether one forecaster’s losses differ from another’s,
unconditionally and conditionally, and whether a forecast is efficient,
nested-better, or encompassed; the model confidence set answers the
many-model version of the same question; conditional forecasts hold
part of a system on a stated path and ask what the rest does; and the
fan chart hands over the quantile bands of any of it.
The package has one rule that every module keeps: everything is
reported per origin, per horizon, and per series before anything is
averaged, and every loss is negatively oriented. A comparison test needs
the loss series and its serial correlation, a calibration record needs
the whole stack of transforms, and a summary that averaged first would
have thrown away exactly what those tests use; the (H, k) means and
the summary tables are conveniences on top of arrays that are the
result. Alignment is the other half of the rule. The tests trust that
origin \(t\) of one series is origin \(t\) of the other and
cannot verify it, so the backtest is where that promise is made and the
tests are where it is spent.
Each module is a leaf: names are imported from the module, not from this package, and every producer takes arrays – predictive paths, outcomes, losses – rather than models, so a forecaster from outside the package is graded on the same terms as one from inside it.
Layout. backtest is the harness,
Backtest, and its record,
BacktestResult, which every
other module can read from. scoring is
DensityScore and
pinball_loss();
calibration is
Calibration with the PIT series
and the Berkowitz test; comparison is
ForecastComparison for
Diebold-Mariano with its corrections and Giacomini-White, and
clark_west(),
mincer_zarnowitz(),
encompassing(), and
giacomini_white() for the forecast
tests; confidence_set is
model_confidence_set();
conditional is
conditional_forecast(); and
fan is
fan_chart(). Model combination, which
pools the density forecasters these tools rank, lives with the Bayesian
workflow in combination.
Example
Backtest two forecasters on one schedule, compare their losses, and read the density forecaster’s calibration at horizon one:
>>> import numpy as np
>>> from cultivars.multivariate.reduced_form.vector_autoregression import VAR
>>> from cultivars.multivariate.large_dim.bayesian import BVAR
>>> y = np.random.default_rng(0).standard_normal((160, 2))
>>> point = backtest.Backtest(y, lambda w: VAR(w, order=1).fit(), start=120).run()
>>> fit = lambda w: BVAR(w, order=1).fit(n_draws=100, seed=0)
>>> density = backtest.Backtest(y, fit, start=120).run(seed=0)
>>> point.n_origins, density.n_draws
(40, 100)
>>> first, second = point.losses(name="y1"), density.losses(name="y1")
>>> comparison.ForecastComparison(first, second).compute().nobs
40
>>> calibration.Calibration(*density.record(1)).compute().pit.shape
(40, 2)
Modules#
Rolling-origin backtests: the harness that produces aligned forecasts and outcomes. |
|
Calibration: does reality land where the densities said it would? |
|
Comparing forecasters: is the loss gap real, and does it persist? |
|
Conditional forecasts: the system's future given stated paths for part of it. |
|
The model confidence set: which forecasters the data cannot rank below the best. |
|
Fan-chart extraction: quantile paths, and nothing else. |
|
Proper scores for density forecasts: what a predictive sample earned. |