Validation¶
Nothing here is asserted on the strength of having been transcribed carefully. Every closed form is checked against something computed a different way, and the checks run in CI.
Published reference values¶
The contingency-table Bayes factors are validated against the worked examples in Jamil et al. [2017], under each of the four Gunel-Dickey sampling schemes.
Example |
Scheme |
Published |
|
|---|---|---|---|
Doll preference [Hraba and Grant, 1970] |
independent multinomial |
log BF₁₀ = 23.03 |
23.0337 |
Simulation, c = 10 |
Poisson |
BF₁₀ = 9.19 |
9.194 |
Simulation, c = 10 |
hypergeometric |
BF₁₀ = 3.04 |
3.043 |
Doll preference, posterior |
— |
median log OR 2.47, 95% CI (1.73, 3.26) |
2.47, (1.73, 3.26) |
The evidence ordering the paper derives — Poisson > joint multinomial > independent multinomial > hypergeometric — is asserted as well, since it must hold for any data.
Independent implementations¶
The two integrals this package shares with existing tools are checked against
those tools directly. pingouin is a validation-only dependency
(pip install bayesplain[validate]), never required at runtime.
Quantity |
Checked against |
Agreement |
|---|---|---|
JZS t-test Bayes factor [Rouder et al., 2009] |
|
6 significant figures, one- and two-sample, across prior widths |
Correlation Bayes factor [Ly et al., 2016] |
|
7 significant figures, across prior widths |
One difference is worth naming. bayesplain evaluates the correlation
marginal likelihood in log space, shifting by the peak of the integrand before
exponentiating. pingouin returns inf or nan for strongly correlated large
samples — including the roughly 2,400-tract case in the bundled ACS data, where
the true log Bayes factor is around 1,125. Both agree wherever both produce a
number.
Self-validation¶
Some quantities can be pinned without any external reference, which is the strongest form of check available:
Marginal likelihoods against brute-force integration. The Beta-Binomial marginal likelihood is compared to
scipy.integrate.quadover the same integral, and to the requirement that it sums to 1 across every possible number of successes.The contingency Bayes factor against quadrature. For a two-column table each marginal likelihood reduces to a one-dimensional integral evaluated to machine precision; for three columns it is checked by Monte Carlo integration over the Dirichlet priors.
The correlation sampling density against its own normalisation. The exact density of an observed correlation must integrate to 1 over (−1, 1) for every ρ and n. It does, to 1 × 10⁻⁶, which pins the transcription of the hypergeometric expression without reference to anyone else’s code.
The independent-multinomial Bayes factor is computed in a form that differs
algebraically from Equation 10 of Jamil et al. [2017] — each row drawing its own
column distribution, rather than the paper’s successive conditioning. Expanding
both in gamma functions, every term involving Γ(RCa) and Γ(y.. + RCa)
cancels and the two agree exactly. The advantage of the form used here is that
it accepts a per-column concentration vector, so a Beta(a, b) prior on a rate
carries into the Bayes factor unchanged.
Behavioural properties¶
Invariances and monotonicities that must hold whatever the formula:
posterior means fall between the prior mean and the maximum-likelihood estimate
interval widths shrink roughly as 1/√n
a wider prior always weakens the Bayes factor for a small observed effect
swapping group order flips the sign of a difference and changes nothing else
results are invariant to relabelling categories, and to reordering rows
a stronger table prior pulls an association toward zero
Pedagogical regression tests¶
Unusual, and the reason for them is practical: if a refactor breaks a lecture,
CI should say so rather than a student. Marked pedagogy and run by default.
the eviction-style two-proportion example yields P ≈ 0.94 while the chi-square p-value exceeds 0.05, and the two intervals agree to within a percentage point
the prior stops mattering as n grows, by at least a factor of five between n = 20 and n = 2000
the credible interval for a mean coincides with the t interval to nine decimal places
in the bundled ACS data, the income-rent correlation at tract level and at block-group level have non-overlapping credible intervals — the modifiable areal unit problem, with the arithmetic to prove it is not sampling noise
in the bundled parcel data, the smallest group tops the ranking until
pool=True, and then does not
Running the checks¶
pytest # everything above except the slow Monte Carlo runs
pytest -m '' # including those
pytest -m validation # only the cross-implementation checks
pytest -m pedagogy # only the numbers a lecture depends on
Full citations for everything referenced above are on the references page.