cultivars.bayes.combination#
Model combination: Bayesian model averaging and predictive stacking.
Two models fitted to the same sample rarely deserve a winner-take-all verdict. A combination weights them and predicts from the mixture \(\sum_m w_m\, p_m(\tilde y \mid y)\), and the two principled ways to choose \(w\) on the simplex answer different questions. Bayesian model averaging takes the posterior model probabilities,
marginal likelihood times prior probability, normalized (Hoeting et al. 1999). It is the coherent answer when one of the models is true, and it concentrates on the best model as the sample grows. When none is true it still concentrates, on the model closest in Kullback-Leibler divergence, which is the known weakness: an average over misspecified models ends up as a single model with extra steps. Stacking takes instead the weights that maximize the held-out log predictive score of the mixture,
over evaluation origins \(t\) (Yao, Vehtari, Simpson & Gelman 2018). It assumes no model is true and does not concentrate: two models that err in different directions keep sharing weight indefinitely, which is why stacking is the combination to use for forecasting. Its price is the held-out densities, one per model per origin, which a rolling-origin backtest produces and nothing else legitimately can.
Two commitments shape the surface. First, the two routes return the same
record, because whatever produced the weights, prediction is the same
act: draw a model in proportion to its weight, then one of that model’s
predictive paths. The routes differ only in the record’s method and
in what its scores column holds, so a consumer – a fan chart, a
backtest – never needs to know which was used. Second, averaging refuses
models whose marginal likelihoods score different numbers of
observations. Models of different autoregressive order condition on
different presamples, and a ratio of their marginal likelihoods is a
ratio of densities of different data, not a Bayes factor; the function
says so and asks for a common presample rather than silently comparing
them.
Layout. bayesian_model_average() and stacking() are the two
constructors. The former collects a
MarginalLikelihoodSelection per model, through
marginal_likelihood() unless the caller
supplies them, and normalizes on the log scale; the latter hands the
(T, M) density matrix to _core._estimators._stacking_weights,
which maximizes the concave objective through a softmax parameterization
of the simplex. Both return a ModelCombinationSelection, the
frozen record in _internals that owns the mixing –
forecast_paths through _core._samplers._mix_predictive_paths, and
the (16th percentile, mean, 84th percentile) forecast the family
convention expects – and the summary table. The gatekeepers
_validate_predictive, _validate_log_density_matrix, and
_validate_names are _core validators shared with the rest of the
forecasting surface.
References
Hoeting, J. A., Madigan, D., Raftery, A. E., & Volinsky, C. T. (1999). Bayesian model averaging: A tutorial. Statistical Science, 14(4), 382-401.
Geweke, J., & Amisano, G. (2011). Optimal prediction pools. Journal of Econometrics, 164(1), 130-141.
Yao, Y., Vehtari, A., Simpson, D., & Gelman, A. (2018). Using stacking to average Bayesian predictive distributions. Bayesian Analysis, 13(3), 917-1007.
Example
Stacking two stand-in models on held-out log densities in which the
first is better at every origin; any object with forecast_paths
can be combined:
>>> import numpy as np
>>> rng = np.random.default_rng(0)
>>> class Toy:
... def __init__(self, level):
... self.level = level
... def forecast_paths(self, steps=8, *, seed=None):
... return np.full((100, steps, 1), self.level)
>>> good = -0.5 * rng.standard_normal(200) ** 2
>>> weights = stacking(
... np.column_stack([good, good - 3.0]), (Toy(0.0), Toy(1.0)), names=("a", "b")
... )
>>> weights.method, round(float(weights.weights[0]), 2)
('stacking', 1.0)
>>> float(weights.forecast_paths(4, seed=0).mean())
0.0