ForecastComparison#

class cultivars.forecast.comparison.ForecastComparison(losses_a, losses_b)[source]

Bases: object

Test a loss differential between two forecasters.

Takes two aligned loss series, one value per evaluation origin, and answers the unconditional question – is the average of \(d_t = L_{A,t} - L_{B,t}\) distinguishable from zero, with a variance that respects the serial correlation multi-step errors carry – twice, as Diebold-Mariano and with the Harvey-Leybourne-Newbold correction, and the conditional question once, as Giacomini-White on the default instruments. The losses can be anything negatively oriented: squared errors, CRPS, log scores, pinball losses, so long as both series were produced by the same loss on the same origins.

Parameters:

  • losses_a (ArrayLike) – (T,) losses of the first forecaster, one per evaluation origin, negatively oriented.

  • losses_b (ArrayLike) – (T,) losses of the second, aligned origin by origin – an alignment this object cannot verify and wholly depends on.

Variables:

  • _losses_a – The validated (T,) first loss series.

  • _losses_b – The validated (T,) second loss series.

Raises:

  • DimensionError – If the series’ shapes disagree.

  • SpecificationError – If the window is too short, or the series are numerically identical and there is no differential to test.

  • NumericalError – If the losses are not finite.

Note

Producing the loss series is deliberately the caller’s job: the rolling-origin loop belongs in driver code or in Backtest, whose record’s losses() gives exactly the aligned series this class takes, because conflating model selection with model evaluation inside a test is how look-ahead bias gets laundered. Fewer than eight origins are refused: a differential’s long-run variance on so short a window has neither power nor reliable size.

See also

References

Diebold, F. X., & Mariano, R. S. (1995). Comparing predictive accuracy. Journal of Business & Economic Statistics, 13(3), 253-263.

Harvey, D., Leybourne, S., & Newbold, P. (1997). Testing the equality of prediction mean squared errors. International Journal of Forecasting, 13(2), 281-291.

Example

Synthetic loss series, then the squared-loss columns of two VAR backtests on one schedule:

>>> import numpy as np
>>> rng = np.random.default_rng(0)
>>> shared = rng.standard_normal(120) ** 2
>>> a = shared + 0.05 * rng.standard_normal(120)
>>> b = shared + 0.4 + 0.05 * rng.standard_normal(120)
>>> verdict = ForecastComparison(a, b).compute()
>>> bool(verdict.mean_differential < 0.0)
True
>>> bool(verdict.hln_pvalue < 0.01)
True
>>> from cultivars.forecast.backtest import Backtest
>>> from cultivars.multivariate.reduced_form.vector_autoregression import VAR
>>> y = np.random.default_rng(1).standard_normal((160, 2))
>>> one = Backtest(y, lambda w: VAR(w, order=1).fit(), horizons=2, start=100).run()
>>> two = Backtest(y, lambda w: VAR(w, order=2).fit(), horizons=2, start=100).run()
>>> first, second = one.losses(horizon=2, name="y1"), two.losses(horizon=2, name="y1")
>>> ForecastComparison(first, second).compute(horizon=2).nobs
59

Private attributes#

_losses_a

_losses_b

On this page