cultivars.bayes.checks#
Prior and posterior predictive checks: the model as a data-generating machine.
A Bayesian model is a machine for generating data, and the two checks in this module run the machine and compare what comes out with what went in. A posterior predictive check simulates replicated data sets from the retained posterior draws and asks whether the fitted model reproduces the features of the sample it was fitted to (Rubin 1984; Gelman, Meng & Stern 1996). A prior predictive check simulates them from the prior instead and shows what the prior deems plausible on the scale of the data before any sample is touched, which is the step of a Bayesian workflow at which a Minnesota tightness or a volatility-of-volatility scale is chosen with the eyes open (Gabry et al. 2019). Both compare discrepancy statistics, per-variable features of a data set, and report where the data sit in the replicated distribution of each as the tail probability
with ties counted as one half. The statistics offered are mean,
sd, min, max, skewness, kurtosis (excess), acf1
(lag-one autocorrelation), and arch1 (lag-one autocorrelation of the
squared demeaned series); the default five – mean, sd, acf1,
kurtosis, arch1 – are the ones whose failure names the
misspecification. A Gaussian innovation cannot match a fat-tailed
kurtosis; a constant covariance cannot match the volatility clustering
in arch1; a mis-specified deterministic term shows in mean.
Two commitments shape the surface. First, the posterior check is presented
as a diagnostic and not as a test: under a correctly specified model the
posterior predictive p-value is concentrated near one half rather than
uniform, because the same data fit the parameters and judge the fit (Meng
1994), so the record flags tails, names the cell, and applies no
multiplicity correction across its m * k cells. Second, the record
keeps statistics and not panels: a check on two hundred replications of a
hundred-variable system is a few kilobytes, whatever the sample length,
and the replicated panels are freed as soon as their statistics are taken.
Layout. PredictiveCheckTest is the frozen record, with the
p-values, the flagged cells, the replicated quantile bands, and the
summary table. replication_check() builds it from an observed panel
and replications of any provenance, and is the whole comparison;
posterior_predictive_check() and prior_predictive_check() are
that function preceded by the simulation and the labelling. The
simulation belongs to the models: a result that can replicate its sample
satisfies ReplicatingResult by exposing
observed and posterior_replications() – the conjugate, Gibbs,
Student-t, and stochastic-volatility BVARs through
_ReplicationMixin, and the
stochastic-volatility posterior on its own – and a model that can draw
from its prior satisfies ReplicatingModel by
exposing prior_replications(): the conjugate and Student-t BVARs with
any dummy observations folded into the prior, the Gibbs BVAR under a
static prior without dummies, and the stochastic-volatility model with
the sampler’s prior hyperparameters. A result may also attach
_replication_notes(), remarks on how it replicates, which the
posterior check carries onto the record. The numerics live in _core:
_discrepancy_statistics evaluates the registry of statistics
column-wise, _predictive_pvalues applies the tie-corrected tail rule,
and _validate_replications and _validate_statistics are the
gatekeepers.
References
Rubin, D. B. (1984). Bayesianly justifiable and relevant frequency calculations for the applied statistician. Annals of Statistics, 12(4), 1151-1172.
Meng, X.-L. (1994). Posterior predictive p-values. Annals of Statistics, 22(3), 1142-1160.
Gelman, A., Meng, X.-L., & Stern, H. (1996). Posterior predictive assessment of model fitness via realized discrepancies. Statistica Sinica, 6(4), 733-760.
Gabry, J., Simpson, D., Vehtari, A., Betancourt, M., & Gelman, A. (2019). Visualization in Bayesian workflow. Journal of the Royal Statistical Society A, 182(2), 389-402.
Gelman, A., Vehtari, A., Simpson, D., Margossian, C. C., Carpenter, B., Yao, Y., Kennedy, L., Gabry, J., Bürkner, P.-C., & Modrák, M. (2020). Bayesian workflow. arXiv:2011.01808.
Example
Fit a conjugate BVAR to a stable system and check that it reproduces the sample’s features:
>>> import numpy as np
>>> from cultivars.multivariate.large_dim.bayesian import BVAR
>>> rng = np.random.default_rng(0)
>>> y = np.zeros((160, 2))
>>> for t in range(1, 160):
... y[t] = 0.6 * y[t - 1] + rng.standard_normal(2)
>>> res = BVAR(y, order=1).fit(n_draws=300, seed=0)
>>> check = posterior_predictive_check(res, seed=0)
>>> check.kind, check.observed.shape, check.replicated.shape
('posterior', (5, 2), (200, 5, 2))
>>> check.adequate()
True
The same model’s prior, before fitting, on the two features the Minnesota prior is about:
>>> prior = prior_predictive_check(BVAR(y, order=1), statistics=["sd", "acf1"], seed=0)
>>> prior.kind, prior.n_replications
('prior', 200)
Classes#
A prior or posterior predictive check: the data against replicated data. |