cultivars.forecast.confidence_set#
The model confidence set: which forecasters the data cannot rank below the best.
A pairwise test says whether one forecaster beats another. With many forecasters the question is different – which of them could be the best – and answering it with pairwise tests multiplies the size distortion. Hansen, Lunde and Nason (2011) answer it the way a confidence interval answers a point estimate: a set of models that contains the best one with probability at least the confidence level,
built by eliminating the worst model while the hypothesis that the remaining ones are equally good, \(\mathbb{E}[L_{t,i} - L_{t,j}] = 0\) for every pair in the set, is rejected. The test statistic is the largest studentized pairwise differential (the range statistic) or the largest studentized deviation from the set average (the max statistic), and its null distribution is taken from a stationary bootstrap of the loss panel (Politis & Romano, 1994) so that the serial correlation of forecast losses is respected rather than assumed away.
Two commitments shape the surface. First, the input is a loss panel,
not a set of models: (T, M) losses aligned origin by origin, which
is what stacking each model’s
losses() from
backtests on one schedule produces. The procedure never sees the
forecasters, so anything that yields a loss series can enter, and the
alignment it depends on is the backtest’s responsibility. Second, the
set is the honest multi-model answer, in the same spirit as the
set-identified impulse responses elsewhere in the package: on a short
or noisy evaluation window it keeps many models, and that is the
finding. Each model carries an MCS p-value, the running maximum of the
elimination p-values up to its own, so that the set at any level is
read off the p-values rather than recomputed.
Layout. model_confidence_set() is the producer and
ModelConfidenceSetSelection the record, the
_ModelConfidenceSetSelection of _internals under its public
name, carrying the p-values, mean losses, elimination order, and the
included / excluded / best readings. The numerics live in
_core: _model_confidence_set runs the elimination loop with the
stationary bootstrap and returns the order and the p-values;
_MCS_ALPHA and _MCS_BOOTSTRAP are the 10% level and replication
count the defaults follow; _validate_names / _variable_names
resolve the model labels.
References
Hansen, P. R., Lunde, A., & Nason, J. M. (2011). The model confidence set. Econometrica, 79(2), 453-497.
Politis, D. N., & Romano, J. P. (1994). The stationary bootstrap. Journal of the American Statistical Association, 89(428), 1303-1313.
White, H. (2000). A reality check for data snooping. Econometrica, 68(5), 1097-1126.
Example
Three loss series, the third a unit worse; the set excludes it and cannot separate the other two:
>>> import numpy as np
>>> rng = np.random.default_rng(0)
>>> losses = rng.standard_normal((150, 3)) ** 2
>>> losses[:, 2] += 1.0
>>> mcs = model_confidence_set(losses, names=("A", "B", "C"), n_bootstrap=300, seed=0)
>>> "C" in mcs.included, mcs.pvalue("C") < 0.1
(False, True)
>>> mcs.included, mcs.best
(('B', 'A'), 'B')
The loss columns of three backtests on one schedule, which is the intended input:
>>> from cultivars.forecast.backtest import Backtest
>>> from cultivars.multivariate.reduced_form.vector_autoregression import VAR
>>> y = np.random.default_rng(1).standard_normal((160, 2))
>>> fits = {p: (lambda w, p=p: VAR(w, order=p).fit()) for p in (1, 2, 3)}
>>> panel = np.column_stack(
... [Backtest(y, f, start=100).run().losses(name="y1") for f in fits.values()]
... )
>>> panel.shape
(60, 3)
>>> model_confidence_set(panel, names=("VAR(1)", "VAR(2)", "VAR(3)"), seed=0).nobs
60
Classes#
Functions#
|
Build the model confidence set from aligned loss series. |