model_confidence_set#

cultivars.forecast.confidence_set.model_confidence_set(losses, *, names=None, alpha=_MCS_ALPHA, statistic='range', n_bootstrap=_MCS_BOOTSTRAP, block_length=None, seed=None)[source]#

Build the model confidence set from aligned loss series.

Hansen, Lunde and Nason’s (2011) sequential procedure on a panel of losses \(L_{t,i}\). With \(d_{ij,t} = L_{t,i} - L_{t,j}\) the pairwise differentials and \(\bar d_{ij}\) their means, the hypothesis that every model in the current set \(\mathcal{M}\) is equally good, \(H_0 : \mathbb{E}[d_{ij,t}] = 0\) for all \(i, j \in \mathcal{M}\), is tested with

\[T_R = \max_{i, j \in \mathcal{M}} \frac{|\bar d_{ij}|}{\sqrt{\widehat{\operatorname{var}}(\bar d_{ij})}} \qquad\text{or}\qquad T_{\max} = \max_{i \in \mathcal{M}} \frac{\bar d_{i\cdot}}{\sqrt{\widehat{\operatorname{var}}(\bar d_{i\cdot})}},\]

where \(\bar d_{i\cdot}\) is model \(i\)’s mean deviation from the set average. The null distribution of the statistic and the variances in its denominator come from a stationary bootstrap of the loss panel (Politis & Romano, 1994), which keeps the serial correlation that overlapping forecast losses carry. While the test rejects at level \(\alpha\), the model with the largest studentized deviation is eliminated and the test repeated on what remains; the model confidence set is the set left when it stops. Each model’s MCS p-value is the running maximum of the elimination p-values up to and including its own, which makes the set at any level exactly the models whose p-value is at least that level.

Parameters:

  • losses (ArrayLike) – (T, M) losses, one column per model and one row per evaluation origin, negatively oriented and aligned origin by origin – BacktestResult.losses(...) from each model’s backtest on the same schedule, stacked as columns.

  • names (Sequence[str] | None) – Model labels; default y1 .. yM.

  • alpha (float) – Level of the set; the default gives the 90% set the authors report.

  • statistic (str) – "range" (largest studentized pairwise differential) or "max" (largest studentized deviation from the set average).

  • n_bootstrap (int) – Stationary-bootstrap replications.

  • block_length (float | None) – Expected block length of the bootstrap; default max(2, round(T ** (1 / 3))).

  • seed (int | Generator | None) – Seed or generator.

Return type:

_ModelConfidenceSetSelection

Returns:

The ModelConfidenceSetSelection.

Raises:

  • DimensionError – If the panel is not (T, M) or the labels do not match.

  • SpecificationError – If the level, statistic, or counts are unusable, or there are fewer than two models or origins.

  • NumericalError – If a loss is not finite.

Note

The columns must be losses of the same origins under the same loss, which is what stacking each model’s losses() from backtests on one schedule guarantees. The set is the honest multi-model answer: on a short or noisy evaluation window it keeps many models, and that is the finding, not a failure of the procedure. At least 100 bootstrap replications are required because the p-values are bootstrap tail frequencies and fewer cannot resolve a 10% level. The default expected block length grows as \(T^{1/3}\), the rate at which the stationary bootstrap’s block should grow for dependent data.

See also

References

Hansen, P. R., Lunde, A., & Nason, J. M. (2011). The model confidence set. Econometrica, 79(2), 453-497.

Politis, D. N., & Romano, J. P. (1994). The stationary bootstrap. Journal of the American Statistical Association, 89(428), 1303-1313.

Example

Three loss series, the third a unit worse on average; the set drops it and keeps the two the window cannot separate:

>>> import numpy as np
>>> rng = np.random.default_rng(0)
>>> losses = rng.standard_normal((150, 3)) ** 2
>>> losses[:, 2] += 1.0
>>> mcs = model_confidence_set(losses, names=("A", "B", "C"), n_bootstrap=300, seed=0)
>>> mcs.included, mcs.excluded, mcs.best
(('B', 'A'), ('C',), 'B')
>>> mcs.elimination_order, mcs.pvalues.round(2)
((2, 0, 1), array([0.22, 1.  , 0.  ]))
>>> mcs.block_length, mcs.nobs
(5.0, 150)

The max statistic with an explicit block length:

>>> alt = model_confidence_set(losses, statistic="max", block_length=3, seed=1)
>>> alt.names, alt.included
(('y1', 'y2', 'y3'), ('y2', 'y1'))
>>> model_confidence_set(losses, n_bootstrap=50)
Traceback (most recent call last):
    ...
cultivars.exceptions.SpecificationError: n_bootstrap must be at least 100; got 50.
On this page