ForecastComparisonResult#

class cultivars.forecast.comparison.ForecastComparisonResult(*, mean_differential, dm_statistic, dm_pvalue, hln_statistic, hln_pvalue, gw_statistic, gw_pvalue, horizon, nobs)[source]#

The verdict on a loss differential, unconditional and conditional.

Losses are negatively oriented, so a negative mean differential favors the first forecaster. Three readings of the same series \(d_t = L_{A,t} - L_{B,t}\) travel together. The Diebold-Mariano statistic

\[\mathrm{DM} = \frac{\bar d}{\sqrt{\hat\sigma^2_{LR} / T}}, \qquad \hat\sigma^2_{LR} = \hat\gamma_0 + 2 \sum_{j=1}^{h-1} \Bigl(1 - \frac{j}{h}\Bigr) \hat\gamma_j,\]

with the Bartlett long-run variance through \(h - 1\) lags, is read against the standard normal; the Harvey-Leybourne-Newbold version scales it by \(\sqrt{(T + 1 - 2h + h(h-1)/T)/T}\) and reads it against \(t_{T-1}\), which corrects the over-rejection of the normal reference in short windows; and the Giacomini-White statistic asks whether \(d_t\) was predictable from a constant and \(d_{t-h}\), the conditional question.

Variables:

  • mean_differential – Average of losses_a - losses_b.

  • dm_statistic – Diebold-Mariano statistic under the standard normal null.

  • dm_pvalue – Its two-sided p-value.

  • hln_statistic – The Harvey-Leybourne-Newbold corrected statistic, referred to a t distribution – the number to trust in the short evaluation windows macroeconomics actually has.

  • hln_pvalue – Its two-sided p-value.

  • gw_statistic – Giacomini-White conditional statistic with a constant and the differential lagged by the horizon – the most recent one known at the origin – as instruments, chi-squared with two degrees of freedom under the null of no conditional predictability; giacomini_white() opens the instrument set.

  • gw_pvalue – Its upper-tail p-value.

  • horizon – The forecast horizon the losses were produced at, which sets the variance’s serial-correlation window.

  • nobs – Evaluation origins.

Note

These are tests about forecasts, not models. Two nested models’ recursive forecasts break the asymptotics – the larger model’s extra parameters inflate its loss under the null and the differential’s limit is not normal – which is Clark-McCracken territory, named and not implemented; clark_west() is the approximately normal substitute for that case. The two-sided p-values answer “do the forecasters differ”; the sign of mean_differential says which way.

See also

References

Diebold, F. X., & Mariano, R. S. (1995). Comparing predictive accuracy. Journal of Business & Economic Statistics, 13(3), 253-263.

Harvey, D., Leybourne, S., & Newbold, P. (1997). Testing the equality of prediction mean squared errors. International Journal of Forecasting, 13(2), 281-291.

Giacomini, R., & White, H. (2006). Tests of conditional predictive ability. Econometrica, 74(6), 1545-1578.

Example

Two loss series sharing a common component, the second worse by 0.4 on average:

>>> import numpy as np
>>> rng = np.random.default_rng(0)
>>> shared = rng.standard_normal(120) ** 2
>>> a = shared + 0.05 * rng.standard_normal(120)
>>> b = shared + 0.4 + 0.05 * rng.standard_normal(120)
>>> verdict = ForecastComparison(a, b).compute()
>>> round(verdict.mean_differential, 3), verdict.nobs, verdict.horizon
(-0.404, 120, 1)
>>> round(verdict.dm_statistic, 1), round(verdict.hln_statistic, 1)
(-58.0, -57.7)
>>> bool(verdict.hln_pvalue < 1e-6), bool(verdict.gw_pvalue < 1e-6)
(True, True)

Attributes#

mean_differential

Average of losses_a - losses_b; negative favors the first forecaster.

dm_statistic

Diebold-Mariano statistic, the mean differential over its long-run standard error.

dm_pvalue

Two-sided standard normal p-value of dm_statistic.

hln_statistic

dm_statistic scaled by the Harvey-Leybourne-Newbold small-sample factor.

hln_pvalue

Two-sided \(t_{T-1}\) p-value of hln_statistic; the one to read in short windows.

gw_statistic

Giacomini-White Wald statistic on a constant and \(d_{t-h}\); \(\chi^2_2\).

gw_pvalue

Upper-tail chi-squared p-value of gw_statistic.

horizon

Forecast horizon behind the losses, \(h\); the long-run variance uses h - 1 lags.

nobs

Evaluation origins, \(T\).

Methods#

summary()

The estimation summary, renderable as text, HTML, or a dataframe.