PredictiveCheckTest#

class cultivars.bayes.checks.PredictiveCheckTest(*, kind, statistics, names, observed, replicated, n_replications, nobs, source, notes=())[source]#

A prior or posterior predictive check: the data against replicated data.

The object posterior_predictive_check(), prior_predictive_check(), and replication_check() return. Each replication \(y^{\text{rep},(r)}\) is a data set of the same shape as the observed panel, simulated from one draw \(\theta^{(r)}\) of the parameters. From the posterior, the question is whether the fitted model reproduces the features of the sample it was fitted to; from the prior, what the prior deems plausible on the scale of the data before any sample touches it. A discrepancy statistic \(T\) is evaluated per variable on the data and on every replication, and the tail probability

\[p_T = \Pr\bigl(T(y^{\text{rep}}) \ge T(y)\bigr) \approx \frac{1}{R} \sum_{r=1}^{R} \mathbb{1}\bigl\{T(y^{\text{rep},(r)}) \ge T(y)\bigr\},\]

locates the data in the replicated distribution, with ties counted as one half so that a statistic the model reproduces exactly reads \(0.5\) rather than \(1\). Near zero or one, the model does not produce data like these in that respect, and the statistic’s name says which respect: a Gaussian innovation cannot match a fat-tailed kurtosis, a constant covariance cannot match the volatility clustering in arch1, a misplaced constant shows in mean.

The posterior version is not a frequentist test. Under a correctly specified model \(p_T\) is concentrated near one half rather than uniform (Meng, 1994), because the same data fit the parameters and judge the fit, so a value in a tail understates the evidence of misspecification rather than overstating it. The prior version is not a test at all, since the data are one draw the prior may or may not cover, and its table is read for the range of the replicated statistics as much as for the tail probability (Gabry et al., 2019): a prior whose 90% band for sd runs from a thousandth to a thousand times the data’s scale is uninformative in a way no p-value shows.

The record is frozen and holds only the statistics, never the replicated panels themselves, so it is small whatever \(R\) and \(T\) were; every derived quantity is recomputed from observed and replicated on access.

Shapes:

R – Replicated data sets, n_replications.
T – Rows of the data set replicated, nobs.
k – Variables, len(names).
m – Discrepancy statistics, len(statistics).

Variables:

kind – "posterior" or "prior", which sets how the summary tells the reader to interpret the tail probabilities.
statistics – Names of the discrepancy statistics, in row order.
names – Variable labels, in column order.
observed – (m, k) statistics of the data.
replicated – (R, m, k) statistics of the replications.
n_replications – Replicated data sets \(R\).
nobs – Rows in the data set replicated, the effective sample.
source – What was checked, for the summary title.
notes – Remarks from the replicating result or model, prepended to the summary’s notes.

Note

The tail probabilities are resolved to \(1 / R\), so with the default 200 replications the smallest non-zero value is \(0.005\), and a p-value of exactly zero means no replication reached the data, not that the probability is zero. The summary says so below 200 replications.

Example

Replications that match the data in every default respect:

>>> import numpy as np
>>> rng = np.random.default_rng(0)
>>> y = rng.standard_normal((200, 2))
>>> rep = rng.standard_normal((300, 200, 2))
>>> check = replication_check(y, rep, names=("a", "b"))
>>> check.statistics
('mean', 'sd', 'acf1', 'kurtosis', 'arch1')
>>> check.pvalues.shape, check.adequate()
((5, 2), True)

Heavy-tailed data against Gaussian replications: the scale and the mean are matched, the kurtosis is not, and the name of the failing statistic is the diagnosis:

>>> heavy = rng.standard_t(3, size=(200, 2))
>>> heavy = (heavy - heavy.mean(axis=0)) / heavy.std(axis=0)
>>> check = replication_check(heavy, rep, names=("a", "b"))
>>> check.extreme()
('kurtosis[a]', 'kurtosis[b]')
>>> check.pvalue("kurtosis", "a")
0.0
>>> check
PredictiveCheckTest(posterior, 300 replications, 5 statistics x 2 variables, 2 in a tail)

Attributes#

`kind`	which distribution the replications came from.
`statistics`	Names of the discrepancy statistics, one per row of `observed`.
`names`	Variable labels, one per column of `observed`, in column order.
`observed`	`(m, k)` discrepancy statistics of the observed panel.
`replicated`	`(R, m, k)` statistics of every replication.
`n_replications`	Replicated data sets \(R\), the leading axis of `replicated`.
`nobs`	Rows in the data set that was replicated, the effective sample.
`source`	What was checked, as it appears in the summary title.
`notes`	Remarks the replicating result or model attached, prepended to the summary's notes.
`pvalues`	`(m, k)` tail probabilities \(\Pr(T(y^{\text{rep}}) \ge T(y))\).

Methods#

`adequate`(*[, level])	Whether no statistic sits in a tail of its replicated distribution.
`extreme`(*[, level])	Labels `statistic[variable]` with a tail probability outside `(level, 1 - level)`.
`pvalue`(statistic[, name])	One statistic's tail probabilities, for every variable or for one.
`replicated_quantiles`([levels])	Quantiles of each replicated statistic across the replications.
`summary`(*[, level])	Render as a table: one row per statistic and variable.

PredictiveCheckTest#

Attributes#

Methods#

API reference

Examples

Engine

Get cultivars