The frequentist twin¶
No result in this package appears without the conventional test that would ordinarily have been run on the same data. The point is not even-handedness for its own sake: it is that nobody should finish a course able to produce a posterior but unable to read a p-value in someone else’s report.
Analysis |
Counterpart it prints |
|---|---|
|
exact binomial test, Wilson score interval |
|
two-proportion z / chi-square, with a Wald interval on a difference or a log-scale interval on a ratio; chi-square test of equal proportions for three or more groups |
|
exact Poisson test, exact (Garwood) interval |
|
exact conditional test for two Poisson rates |
|
Pearson chi-square test of independence |
|
chi-square goodness-of-fit test |
|
one-sample t-test (on the log scale with |
|
Welch’s t-test (or pooled Student’s, or paired) |
|
Pearson r, Fisher-z interval |
|
t-test on the least-squares slope |
|
one-way ANOVA |
They usually agree on the numbers¶
import numpy as np
import bayesplain as bp
rng = np.random.default_rng(0)
x = rng.normal(50, 12, 40)
res = bp.mean(x)
print("credible interval :", np.round(res.interval(kind="eti"), 6))
print("confidence interval:", np.round(res.frequentist.interval, 6))
credible interval : [46.215373 52.333964]
confidence interval: [46.215373 52.333964]
Identical to six decimal places, and not by luck: under the reference prior the posterior for a mean is exactly the t distribution the confidence interval is built from. The arithmetic is the same. What differs is the sentence you are allowed to say afterwards.
print(res.translate())
THE QUANTITY: average (from 40 observations)
What the posterior says
Given this data and the stated prior, there is a 95% probability the
value lies in the range 46.22 to 52.33, and a 100% probability it is
greater than zero. Both are statements about the quantity itself.
What the one-sample t-test says
If the true mean were exactly 0, data at least this extreme would turn
up less than once in a million studies.
It is not the probability that the null is true, not the probability
this finding is wrong, and it says nothing about how large the effect
is. p = 6.7e-30 is a statement about data given a hypothesis, not about
a hypothesis given data.
Across many repetitions of this study, 95% of intervals built this way
would contain the true value. It does not say there is a 95% chance the
true value lies in the range 46.22 to 52.33 — that sentence is only
available for a credible interval.
How they line up
Both point the same way here, so the choice between them is not
about the conclusion but about what you can say. 'Significant'
licenses a claim about the data; the posterior licenses a claim
about the size of the effect, which is what a recommendation
actually needs.
And sometimes they seem not to¶
res = bp.compare_proportions(
[34, 51], [220, 240], labels=["District A", "District B"]
)
print(res.summary())
==========================================================================
difference in rate (District B − District A)
==========================================================================
BAYESIAN — what the data say about the quantity itself
best estimate (median) 5.7 percentage points
95% credible interval −1.2 to 12.9 percentage points (HDI)
P(District B higher) 0.945
Read: District B is higher than District A with 94% probability; the
best estimate of the gap is 5.7 percentage points, and the data
are consistent with anything from −1.2 to 12.9 percentage points
(95% credible interval).
FREQUENTIST — two-proportion z-test (equivalently chi-square, 1 df)
chi-square (1 df) 2.559
p-value 0.1096 (not significant at 0.05)
95% confidence interval −1.2 to 12.8 percentage points (Wald)
Read: If the two rates were identical, data at least this extreme would
turn up 11.0% of the time.
--------------------------------------------------------------------
Why they look like they disagree
They are answering different questions. The test asks how
surprising this data would be in a world where the two rates were
identical, and 11.0% is not surprising enough to clear the
conventional bar. The posterior asks how plausible each possible
difference is given the data you have, and puts 94% of that
plausibility on one side. Neither is wrong. Only the second one is
an input to a decision.
--------------------------------------------------------------------------
assumption prior: uninformed — Beta(1, 1)
precision 100,000 independent draws; Monte Carlo error on the mean
is ±0.00011. These are exact samples, not a Markov chain.
next .probability('>', x) .decide(rope=(lo, hi)) .sensitivity()
==========================================================================
“Not significant” and “94% probability District B is higher” are both correct statements about this data. They are answers to different questions. The test asks how surprising the data would be in a world where the rates match; the posterior asks how plausible each difference is given the data in hand. Only the second is an input to a decision.
What each one claims, in words¶
Every twin can state its own case:
twin = res.frequentist
print(twin.claims())
print()
print(twin.disclaims())
print()
print(twin.interval_claims())
If the two rates were identical, data at least this extreme would turn up 11.0% of the time.
It is not the probability that the null is true, not the probability this finding is wrong, and it says nothing about how large the effect is. p = 0.110 is a statement about data given a hypothesis, not about a hypothesis given data.
Across many repetitions of this study, 95% of intervals built this way would contain the true value. It does not say there is a 95% chance the true value lies in the range -0.01247 to 0.1284 — that sentence is only available for a credible interval.
That last one is the confidence-interval hook. Most people — including plenty of published researchers — read a 95% CI as “95% chance the true value is in here”. That is wrong, and the error is durable because it is the sentence people actually want to say. A credible interval is what finally lets them say it correctly.
Using the twins on their own¶
They are ordinary functions, usable without any Bayesian machinery:
from bayesplain import frequentist as fq
fq.two_proportions([34, 51], [220, 240])
FrequentistTwin(test='two-proportion z-test (equivalently chi-square, 1 df)', statistic_name='chi-square', statistic=2.559272727272727, pvalue=0.10964901471069712, dof=1.0, estimate=0.05795454545454545, interval=(-0.012472826386950173, 0.1283819172960411), interval_level=0.95, interval_method='Wald', null_statement='the two rates were identical')
rng = np.random.default_rng(1)
groups = [rng.normal(m, 3, 40) for m in (10, 12, 11)]
fq.one_way_anova(groups)
FrequentistTwin(test='one-way ANOVA', statistic_name='F', statistic=3.752590207089863, pvalue=0.02632744587262931, dof=2.0, estimate=0.06028006536927165, interval=None, interval_level=0.95, interval_method='none reported for eta-squared', null_statement='every group had the same mean')
Where the Bayesian version genuinely does more¶
Two places, both worth a class session.
Effect size with uncertainty. A chi-square test gives you a statistic and a p-value; ask how strong the association is and it offers a point estimate with nothing attached. Drawing from the Dirichlet posterior gives that quantity a full distribution.
import pandas as pd
parcels = bp.datasets.load_parcels()
table = pd.crosstab(parcels.construction, parcels.condition)
res = bp.contingency(table)
lo, hi = res.interval()
print(f"Cramér's V: {res.point():.3f}, 95% credible interval {lo:.3f} to {hi:.3f}")
print(f"chi-square point estimate of V: {res.frequentist.estimate:.3f} (no interval available)")
Cramér's V: 0.078, 95% credible interval 0.064 to 0.092
chi-square point estimate of V: 0.073 (no interval available)
Sparse tables. The chi-square approximation needs roughly five expected cases per cell and misbehaves below that. The posterior does not degrade; it simply widens, which is the honest response to thin data.
sparse = bp.contingency([[1, 20], [3, 18]])
for note in sparse.notes:
print("-", note)
- 2 of 4 cells have expected counts below 5, where the chi-square approximation is unreliable. The posterior is unaffected — it just gets wider, which is the honest response to thin data.