cultivars.forecast.comparison

Comparing forecasters: is the loss gap real, and does it persist?

Diebold-Mariano (1995) asks the plain question – is the average loss differential \(d_t = L_{A,t} - L_{B,t}\) between two forecast series distinguishable from zero –

\[\mathrm{DM} = \frac{\bar d}{\sqrt{\hat\sigma^2_{LR} / T}}, \qquad \hat\sigma^2_{LR} = \hat\gamma_0 + 2 \sum_{j=1}^{h-1} \Bigl(1 - \frac{j}{h}\Bigr)\hat\gamma_j,\]

with a variance that respects the serial correlation multi-step forecast errors mechanically carry, and the Harvey-Leybourne-Newbold small-sample correction always applied alongside, because evaluation windows in macroeconomics are short. Giacomini-White (2006) sharpens the question to a conditional one: not “who was better on average” but “was the gap predictable from what we knew at the time,” which is the version that matters for choosing a forecaster going forward. Around those two sit the tests that read a forecast rather than a loss: Clark-West for the nested case Diebold-Mariano cannot handle, Mincer-Zarnowitz for whether a forecast could be recalibrated against itself, and encompassing for whether one forecast already holds what another knows.

Two commitments shape the surface. First, the inputs are aligned series, one value per evaluation origin – losses for the comparison tests, point forecasts and outcomes for the forecast tests – and producing them is deliberately not this module’s job. The rolling-origin loop belongs in Backtest, whose record hands over exactly these series through losses() and its point and realized arrays, or in the caller’s own driver code; conflating model selection with model evaluation inside a test is how look-ahead bias gets laundered, so the alignment is trusted here, never constructed. Second, two standard abuses are warned against rather than silently permitted. The comparison tests are about forecasts, not models: comparing nested models’ recursive forecasts breaks the Diebold-Mariano asymptotics (Clark-McCracken territory, named and not implemented), and clark_west() is the approximately normal substitute. And the Giacomini-White framework wants rolling-window estimation, whose finite memory is what makes its null well-posed.

Layout. ForecastComparison runs Diebold-Mariano, its HLN correction, and Giacomini-White on the default instruments together and returns a ForecastComparisonResult; giacomini_white() opens the instrument set and returns a GiacominiWhiteTest; clark_west(), mincer_zarnowitz(), and encompassing() return their own records, all four of which share the _ForecastComparisonTest base of _internals for the origin count, horizon, and verdict header. The numerics live in _core: _validate_aligned_series checks the alignment, horizon, and the _MIN_COMPARISON_ORIGINS floor of eight; _bartlett_long_run_variance is the HAC variance every statistic divides by; _conditional_instruments builds the Giacomini-White design with the horizon-aware lag and _giacomini_white its Wald statistic; and _clark_west, _mincer_zarnowitz, and _forecast_encompassing are the three forecast tests.

References

Diebold, F. X., & Mariano, R. S. (1995). Comparing predictive accuracy. Journal of Business & Economic Statistics, 13(3), 253-263.

Harvey, D., Leybourne, S., & Newbold, P. (1997). Testing the equality of prediction mean squared errors. International Journal of Forecasting, 13(2), 281-291.

Giacomini, R., & White, H. (2006). Tests of conditional predictive ability. Econometrica, 74(6), 1545-1578.

Clark, T. E., & West, K. D. (2007). Approximately normal tests for equal predictive accuracy in nested models. Journal of Econometrics, 138(1), 291-311.

West, K. D. (2006). Forecast evaluation. In G. Elliott, C. W. J. Granger, & A. Timmermann (Eds.), Handbook of Economic Forecasting (Vol. 1, pp. 99-134). Elsevier.

Example

Two VAR orders backtested on one schedule, compared on their one-step squared losses and, since the orders nest, by Clark-West:

>>> import numpy as np
>>> from cultivars.forecast.backtest import Backtest
>>> from cultivars.multivariate.reduced_form.vector_autoregression import VAR
>>> y = np.random.default_rng(1).standard_normal((160, 2))
>>> small = Backtest(y, lambda w: VAR(w, order=1).fit(), start=100).run()
>>> large = Backtest(y, lambda w: VAR(w, order=3).fit(), start=100).run()
>>> verdict = ForecastComparison(small.losses(name="y1"), large.losses(name="y1")).compute()
>>> verdict.nobs, bool(verdict.mean_differential < 0.0)
(60, True)
>>> nested = clark_west(small.realized[:, 0, 0], small.point[:, 0, 0], large.point[:, 0, 0])
>>> nested.nobs, bool(nested.reject())
(60, False)

Classes

ClarkWestTest	Verdict of the Clark-West (2007) test that a nesting model forecasts better.
EncompassingTest	Verdict of the forecast encompassing test: does A already contain what B knows?
ForecastComparison	Test a loss differential between two forecasters.
ForecastComparisonResult	The verdict on a loss differential, unconditional and conditional.
GiacominiWhiteTest	Verdict of the Giacomini-White (2006) test of conditional predictive ability.
MincerZarnowitzTest	Verdict of the Mincer-Zarnowitz efficiency regression y = a + b f.