The frequentist twin

No result in this package appears without the conventional test that would ordinarily have been run on the same data. The point is not even-handedness for its own sake: it is that nobody should finish a course able to produce a posterior but unable to read a p-value in someone else’s report.

Analysis

Counterpart it prints

proportion

exact binomial test, Wilson score interval

compare_proportions

two-proportion z / chi-square, with a Wald interval on a difference or a log-scale interval on a ratio; chi-square test of equal proportions for three or more groups

rate

exact Poisson test, exact (Garwood) interval

compare_rates

exact conditional test for two Poisson rates

contingency

Pearson chi-square test of independence

categories

chi-square goodness-of-fit test

mean

one-sample t-test (on the log scale with log=True)

compare_means

Welch’s t-test (or pooled Student’s, or paired)

correlation

Pearson r, Fisher-z interval

regression

t-test on the least-squares slope

compare_groups

one-way ANOVA

They usually agree on the numbers

import numpy as np
import bayesplain as bp

rng = np.random.default_rng(0)
x = rng.normal(50, 12, 40)

res = bp.mean(x)
print("credible interval :", np.round(res.interval(kind="eti"), 6))
print("confidence interval:", np.round(res.frequentist.interval, 6))
credible interval : [46.215373 52.333964]
confidence interval: [46.215373 52.333964]

Identical to six decimal places, and not by luck: under the reference prior the posterior for a mean is exactly the t distribution the confidence interval is built from. The arithmetic is the same. What differs is the sentence you are allowed to say afterwards.

print(res.translate())
THE QUANTITY: average (from 40 observations)

What the posterior says
  Given this data and the stated prior, there is a 95% probability the
  value lies in the range 46.22 to 52.33, and a 100% probability it is
  greater than zero. Both are statements about the quantity itself.

What the one-sample t-test says
  If the true mean were exactly 0, data at least this extreme would turn
  up less than once in a million studies.
  It is not the probability that the null is true, not the probability
  this finding is wrong, and it says nothing about how large the effect
  is. p = 6.7e-30 is a statement about data given a hypothesis, not about
  a hypothesis given data.
  Across many repetitions of this study, 95% of intervals built this way
  would contain the true value. It does not say there is a 95% chance the
  true value lies in the range 46.22 to 52.33 — that sentence is only
  available for a credible interval.

How they line up
  Both point the same way here, so the choice between them is not
  about the conclusion but about what you can say. 'Significant'
  licenses a claim about the data; the posterior licenses a claim
  about the size of the effect, which is what a recommendation
  actually needs.

And sometimes they seem not to

res = bp.compare_proportions(
    [34, 51], [220, 240], labels=["District A", "District B"]
)
print(res.summary())
==========================================================================
 difference in rate (District B − District A)
==========================================================================

 BAYESIAN — what the data say about the quantity itself

   best estimate (median)    5.7 percentage points
   95% credible interval     −1.2 to 12.9 percentage points  (HDI)
   P(District B higher)      0.945

   Read: District B is higher than District A with 94% probability; the
         best estimate of the gap is 5.7 percentage points, and the data
         are consistent with anything from −1.2 to 12.9 percentage points
         (95% credible interval).

 FREQUENTIST — two-proportion z-test (equivalently chi-square, 1 df)

   chi-square (1 df)         2.559
   p-value                   0.1096  (not significant at 0.05)
   95% confidence interval   −1.2 to 12.8 percentage points  (Wald)

   Read: If the two rates were identical, data at least this extreme would
         turn up 11.0% of the time.

   --------------------------------------------------------------------

   Why they look like they disagree
     They are answering different questions. The test asks how
     surprising this data would be in a world where the two rates were
     identical, and 11.0% is not surprising enough to clear the
     conventional bar. The posterior asks how plausible each possible
     difference is given the data you have, and puts 94% of that
     plausibility on one side. Neither is wrong. Only the second one is
     an input to a decision.

--------------------------------------------------------------------------
 assumption   prior: uninformed — Beta(1, 1)
 precision    100,000 independent draws; Monte Carlo error on the mean
              is ±0.00011. These are exact samples, not a Markov chain.
 next         .probability('>', x)  .decide(rope=(lo, hi))  .sensitivity()
==========================================================================

“Not significant” and “94% probability District B is higher” are both correct statements about this data. They are answers to different questions. The test asks how surprising the data would be in a world where the rates match; the posterior asks how plausible each difference is given the data in hand. Only the second is an input to a decision.

What each one claims, in words

Every twin can state its own case:

twin = res.frequentist
print(twin.claims())
print()
print(twin.disclaims())
print()
print(twin.interval_claims())
If the two rates were identical, data at least this extreme would turn up 11.0% of the time.

It is not the probability that the null is true, not the probability this finding is wrong, and it says nothing about how large the effect is. p = 0.110 is a statement about data given a hypothesis, not about a hypothesis given data.

Across many repetitions of this study, 95% of intervals built this way would contain the true value. It does not say there is a 95% chance the true value lies in the range -0.01247 to 0.1284 — that sentence is only available for a credible interval.

That last one is the confidence-interval hook. Most people — including plenty of published researchers — read a 95% CI as “95% chance the true value is in here”. That is wrong, and the error is durable because it is the sentence people actually want to say. A credible interval is what finally lets them say it correctly.

Using the twins on their own

They are ordinary functions, usable without any Bayesian machinery:

from bayesplain import frequentist as fq

fq.two_proportions([34, 51], [220, 240])
FrequentistTwin(test='two-proportion z-test (equivalently chi-square, 1 df)', statistic_name='chi-square', statistic=2.559272727272727, pvalue=0.10964901471069712, dof=1.0, estimate=0.05795454545454545, interval=(-0.012472826386950173, 0.1283819172960411), interval_level=0.95, interval_method='Wald', null_statement='the two rates were identical')
rng = np.random.default_rng(1)
groups = [rng.normal(m, 3, 40) for m in (10, 12, 11)]
fq.one_way_anova(groups)
FrequentistTwin(test='one-way ANOVA', statistic_name='F', statistic=3.752590207089863, pvalue=0.02632744587262931, dof=2.0, estimate=0.06028006536927165, interval=None, interval_level=0.95, interval_method='none reported for eta-squared', null_statement='every group had the same mean')

Where the Bayesian version genuinely does more

Two places, both worth a class session.

Effect size with uncertainty. A chi-square test gives you a statistic and a p-value; ask how strong the association is and it offers a point estimate with nothing attached. Drawing from the Dirichlet posterior gives that quantity a full distribution.

import pandas as pd

parcels = bp.datasets.load_parcels()
table = pd.crosstab(parcels.construction, parcels.condition)

res = bp.contingency(table)
lo, hi = res.interval()
print(f"Cramér's V: {res.point():.3f}, 95% credible interval {lo:.3f} to {hi:.3f}")
print(f"chi-square point estimate of V: {res.frequentist.estimate:.3f} (no interval available)")
Cramér's V: 0.078, 95% credible interval 0.064 to 0.092
chi-square point estimate of V: 0.073 (no interval available)

Sparse tables. The chi-square approximation needs roughly five expected cases per cell and misbehaves below that. The posterior does not degrade; it simply widens, which is the honest response to thin data.

sparse = bp.contingency([[1, 20], [3, 18]])
for note in sparse.notes:
    print("-", note)
- 2 of 4 cells have expected counts below 5, where the chi-square approximation is unreliable. The posterior is unaffected — it just gets wider, which is the honest response to thin data.