How to estimate Poisson origin–destination flow models

This guide fits SARPoissonFlow(separable=True) to origin–destination counts and decides between Poisson and negative binomial observation noise for the same spatial structure.

Use it when your flows are counts, you want a correctly specified count likelihood rather than a log-transformed Gaussian, and you need to know whether the extra dispersion parameter of a negative binomial is buying you anything.

The Poisson flow models sample by auxiliary-mixture Gibbs (Frühwirth-Schnatter & Wagner 2006): Poisson inter-arrival times are augmented and \(-\log\tau\) is approximated by a ten-component normal mixture, which leaves a conditionally Gaussian model that the existing SAR machinery — the same \(\rho\) step, the same sparse solves, the same log-determinant — already serves.

Bayesian Poisson is not PPML

Santos Silva & Tenreyro’s Poisson pseudo-maximum-likelihood estimator is a quasi-MLE: its point estimates are consistent under a correct conditional mean regardless of the true variance, and the inferential work is done by sandwich standard errors. A Bayesian posterior under a knowingly misspecified Poisson gives you the right point estimates but credible intervals that are too narrow when the data are overdispersed — nothing in the posterior plays the role of the sandwich.

To carry the actual robustness property you would need a weighted-likelihood bootstrap or a Gibbs posterior with a sandwich-matched learning rate (Müller 2013; Syring & Martin 2019). Neither is implemented here, so this model is family = "poisson" — a correctly specified Poisson, not PPML. Read the comparison below as a specification question, not as a robust alternative to one.

Simulate flow counts

simulate_sar_poisson_flow(separable=True) draws counts from \(y_{ij}\sim\text{Poisson}(\mu_{ij})\) with \(\log\boldsymbol\mu = A(\rho_d,\rho_o,\rho_w)^{-1}X\beta\) and the separable restriction \(\rho_w = -\rho_d\rho_o\). With \(n = 25\) regions there are \(N = n^2 = 625\) flows.

import time

import arviz as az
import numpy as np
import pandas as pd

from neighbayes.dgp import simulate_sar_poisson_flow
from neighbayes.models import SARNegBinFlow, SARPoissonFlow

RHO_D, RHO_O = 0.35, 0.25

data = simulate_sar_poisson_flow(n=25, rho_d=RHO_D, rho_o=RHO_O, seed=7, separable=True)
y, X, G = data["y_vec"], data["X"], data["G"]

print(f"flows      : {y.size}")
print(f"mean count : {y.mean():.2f}")
print(f"variance   : {y.var():.2f}")
print(f"zeros      : {(y == 0).mean():.1%}")
flows      : 625
mean count : 8.12
variance   : 51.81
zeros      : 4.5%
/home/runner/micromamba/envs/test/lib/python3.14/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

Note the variance-to-mean ratio. These counts are conditionally Poisson by construction, yet marginally they look badly overdispersed, because the spatial filter \(A^{-1}\) spreads \(\mu_{ij}\) over orders of magnitude across cells. Marginal overdispersion is not evidence that you need a negative binomial — in flow data it is the ordinary signature of a gravity mean with spatial feedback. The question has to be settled by fitting both.

Fit the Poisson flow model

The interface is the shared flow-model interface: pass the vectorized flow counts, the \(N\times p\) origin–destination design matrix, and the regional weights graph on \(n\) units. Poisson flow models are Gibbs-only — fit(sampler="nuts") raises.

model_pois = SARPoissonFlow(y, X, G, separable=True, col_names=data["col_names"])

start = time.perf_counter()
idata_pois = model_pois.fit(
    draws=1000, tune=500, chains=2, random_seed=11, progressbar=False
)
secs_pois = time.perf_counter() - start

az.summary(idata_pois, var_names=["rho_d", "rho_o", "rho_w"])
mean sd eti89_lb eti89_ub ess_bulk ess_tail r_hat mcse_mean mcse_sd
rho_d 0.316 0.05 0.24 0.39 474 614 1.00 0.0023 0.0012
rho_o 0.231 0.057 0.14 0.32 346 474 1.00 0.0031 0.0015
rho_w -0.073 0.022 -0.11 -0.04 384 634 1.00 0.0011 0.00052

rho_w is deterministic here — the separable model pins it to \(-\rho_d\rho_o\) rather than sampling it — so it carries no independent information. It is reported because the impacts decomposition needs all three.

Check recovery against the truth:

post = idata_pois.posterior
print(f"rho_d: {float(post['rho_d'].mean()):.3f}  (true {RHO_D})")
print(f"rho_o: {float(post['rho_o'].mean()):.3f}  (true {RHO_O})")
rho_d: 0.316  (true 0.35)
rho_o: 0.231  (true 0.25)

Read the spillovers

impacts() gives the LeSage–Thomas-Agnan decomposition of each covariate into origin, destination, intra-regional and network components.

model_pois.impacts().round(4)
mean ci_lower ci_upper bayes_pvalue
predictor side effect
x0 dest origin 0.0079 0.0045 0.0128 0.0
x1 dest origin 0.0069 0.0033 0.0118 0.0
x0 dest destination 0.4019 0.3668 0.4379 0.0
x1 dest destination 0.3898 0.3551 0.4223 0.0
x0 dest intraunit 0.0183 0.0128 0.0241 0.0
x1 dest intraunit 0.0159 0.0101 0.0216 0.0
x0 dest network 0.1735 0.1099 0.2475 0.0
x1 dest network 0.1687 0.1044 0.2448 0.0
x0 dest total 0.6017 0.5341 0.6797 0.0
x1 dest total 0.5813 0.5020 0.6706 0.0
x0 orig origin 0.4515 0.4104 0.4951 0.0
x1 orig origin 0.4624 0.4312 0.4972 0.0
x0 orig destination 0.0055 0.0023 0.0091 0.0
x1 orig destination 0.0056 0.0023 0.0092 0.0
x0 orig intraunit 0.0188 0.0171 0.0206 0.0
x1 orig intraunit 0.0193 0.0180 0.0207 0.0
x0 orig network 0.1320 0.0542 0.2177 0.0
x1 orig network 0.1347 0.0561 0.2206 0.0
x0 orig total 0.6078 0.4977 0.7300 0.0
x1 orig total 0.6220 0.5266 0.7252 0.0

Decide between Poisson and negative binomial

Fit SARNegBinFlow(separable=True) to the same counts, the same design, the same weights. Only the observation model differs — the negative binomial adds a free dispersion parameter alpha.

model_nb = SARNegBinFlow(y, X, G, separable=True, col_names=data["col_names"])

start = time.perf_counter()
idata_nb = model_nb.fit(
    draws=1000, tune=500, chains=2, random_seed=11, progressbar=False
)
secs_nb = time.perf_counter() - start

az.summary(idata_nb, var_names=["rho_d", "rho_o", "alpha"])
mean sd eti89_lb eti89_ub ess_bulk ess_tail r_hat mcse_mean mcse_sd
rho_d 0.31 0.048 0.23 0.39 1304 1162 1.00 0.0013 0.0011
rho_o 0.224 0.06 0.13 0.32 1473 1193 1.00 0.0016 0.0013
alpha 126 1.23e+02 50 290 1843 1384 1.00 3 30

Look at alpha before anything else. Under the negative binomial, \(\operatorname{Var}(y) = \mu + \mu^2/\alpha\), so \(\alpha \to \infty\) is the Poisson. A posterior mean in the hundreds with a standard deviation of the same order is the model telling you it cannot locate the dispersion parameter, because there is no conditional overdispersion to locate. That is the first piece of evidence for the simpler likelihood.

Its converse is worth knowing too: as \(\alpha\) grows, the Pólya–Gamma augmentation the negative binomial sampler relies on degenerates — the working precision \(E[\omega]\) diverges while the marginal Fisher information stays at \(\mu\), and effective sample size collapses. Fixing \(\alpha\) at a large value to “approximate” a Poisson is therefore not a shortcut; it is the one regime where that sampler fails. The auxiliary-mixture scheme exists to reach the Poisson directly instead.

Do the two posteriors agree?

Two independently implemented samplers landing on the same \(\rho\) posterior is mutual validation. Disagreement beyond Monte Carlo error is a specification signal, not a rounding difference.

rows = []
for name in ("rho_d", "rho_o"):
    rows.append(
        {
            "parameter": name,
            "truth": {"rho_d": RHO_D, "rho_o": RHO_O}[name],
            "poisson": float(idata_pois.posterior[name].mean()),
            "negbin": float(idata_nb.posterior[name].mean()),
            "poisson sd": float(idata_pois.posterior[name].std()),
            "negbin sd": float(idata_nb.posterior[name].std()),
        }
    )
pd.DataFrame(rows).set_index("parameter").round(4)
truth poisson negbin poisson sd negbin sd
parameter
rho_d 0.35 0.3160 0.3097 0.0497 0.0480
rho_o 0.25 0.2315 0.2242 0.0567 0.0599

What does each cost?

Sampling efficiency is effective sample size per second, not draws per second. The two likelihoods have different per-sweep costs — the auxiliary-mixture scheme draws two latent inter-arrival times per observation, the Pólya–Gamma scheme draws one mixing weight — so equal draws are not equal work.

rows = []
for label, idata, secs in (
    ("Poisson", idata_pois, secs_pois),
    ("NegBin", idata_nb, secs_nb),
):
    ess = az.ess(idata, var_names=["rho_d", "rho_o"])
    rhat = az.rhat(idata, var_names=["rho_d", "rho_o"])
    rows.append(
        {
            "model": label,
            "seconds": secs,
            "ess rho_d": float(ess["rho_d"]),
            "ess rho_o": float(ess["rho_o"]),
            "ess/sec rho_d": float(ess["rho_d"]) / secs,
            "max rhat": float(max(rhat["rho_d"], rhat["rho_o"])),
        }
    )
pd.DataFrame(rows).set_index("model").round(3)
seconds ess rho_d ess rho_o ess/sec rho_d max rhat
model
Poisson 31.658 474.201 346.797 14.979 1.004
NegBin 3.914 1304.880 1473.232 333.374 1.001

Neither likelihood dominates on speed, and which one comes out ahead shifts with \(n\), with the count level, and with how much of the mass sits near zero — so measure it on your own data rather than carrying a rule of thumb. Efficiency is the tiebreaker, not the criterion. Both samplers here recover the same \(\rho\) posterior with \(\hat R\) at 1.00, so nothing about the science turns on the choice; what should decide it is whether the observation model is right, which is the predictive check below.

What LOO can and cannot settle here

Both models now store the true pointwise log-pmf of the observed counts, so az.loo puts them on one scale and the elpd difference between them is meaningful. Until recently they did not: every negative binomial site stored the NB2 log-pmf without its \(-\log\Gamma(y+1)\) normalizing term, which shifted elpd by \(+\sum_i \log(y_i!)\) — a draw-independent constant, invisible to any shape or finiteness check, large enough to drive the total positive. A positive elpd for discrete data was the tell. That constant is now included at every storage site, and pinned against scipy.stats.nbinom.logpmf by test.

What the comparison still cannot settle is spatial structure. The stored likelihood is conditional on the reduced-form \(\eta\), so leaving one flow out does not leave out its neighbours’ influence on that flow’s own linear predictor; elpd is not a clean out-of-sample quantity under spatial dependence. Read it as a verdict on the observation model — Poisson against negative binomial at fixed spatial structure, which is exactly the question here — and not as a way to choose \(\rho\) parameterizations. Weigh it alongside posterior agreement, sampling efficiency, and the predictive check in count space below.

Posterior predictive check in count space

The honest specification test is whether replicated counts reproduce features of the observed data that the likelihood did not fit directly — here the variance-to-mean ratio and the share of zeros.

y_rep = model_pois.posterior_predictive(random_seed=0)

obs = {"var/mean": y.var() / y.mean(), "zeros": (y == 0).mean(), "max": y.max()}
rep = {
    "var/mean": np.mean(y_rep.var(axis=1) / y_rep.mean(axis=1)),
    "zeros": np.mean((y_rep == 0).mean(axis=1)),
    "max": np.mean(y_rep.max(axis=1)),
}
pd.DataFrame({"observed": obs, "replicated (mean)": rep}).round(3)
observed replicated (mean)
var/mean 6.383 6.497
zeros 0.045 0.041
max 43.000 47.018

If the replicated variance-to-mean ratio and zero share bracket the observed values, the Poisson is doing its job and the negative binomial’s dispersion parameter is fitting noise. If replicated counts are systematically tighter than observed — too little spread, too few zeros — that is genuine excess dispersion and the negative binomial earns its extra parameter.

For real trade and migration flows, Santos Silva & Tenreyro (2011) argue that zero inflation is usually the wrong fix for excess zeros: the zeros in gravity data are ordinary Poisson zeros from small \(\mu_{ij}\), not a separate structural process. Running the comparison above on your own data is the way to check that claim rather than assume it.

When you want all three \(\rho\)

SARPoissonFlow exposes the unrestricted three-parameter model, with \(\rho_w\) free rather than pinned to \(-\rho_d\rho_o\).

Warning

The unrestricted parameterization is weakly identified at moderate \(n\) for both likelihoods. \(\rho_d\), \(\rho_o\) and \(\rho_w\) trade off along a nearly flat ridge that one-at-a-time slice updates cannot traverse: at \(n = 36\) (\(N = 1296\)), effective sample size falls to single digits out of 2000 draws with \(\hat R \approx 1.4\), for this sampler and for SARNegBinFlow alike.

This is weak identification, not a sampler defect — the profile likelihood is genuinely flat along the ridge, and the samplers are exploring it correctly. Tuning will not fix it. Prefer the separable SARPoissonFlow unless \(\rho_w\) is itself the object of interest.

See also