How to fit zero-inflated SAR negative binomial models¶
Count data with more zeros than a negative binomial can account for, where the
excess zeros come from a separate process — units that are inactive
altogether, not merely low-intensity. SARZINB fits the zero and count
parts jointly, each with its own spatial structure and its own weights
matrix, and decomposes the observed zeros between the two.
The unmarked SARZINB is the latent model, each part propagating its own
disturbance; this guide fits the reduced form, SARZINB(latent=False), whose
spatial lags act on the linear predictors only. Both are Gibbs-only. Equations and constructor arguments are in
Supported Models.
Important
Check that you need zero inflation Excess zeros are often ordinary count zeros from a small mean, not a separate process — Santos Silva & Tenreyro (2011) argue exactly this for trade flows. Fit a plain negative binomial first and compare before reaching for this model.
import arviz as az
import matplotlib.pyplot as plt
import numpy as np
from neighbayes.dgp import simulate_sar_zinb
from neighbayes.models import SARZINB
az.style.use("arviz-darkgrid")
plt.rcParams["figure.figsize"] = (10, 6)
/home/runner/micromamba/envs/test/lib/python3.14/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
from .autonotebook import tqdm as notebook_tqdm
Simulate a known process¶
The simulate_sar_zinb function generates data from a two-part spatial model:
Zero part: a reduced-form SAR logit with parameter
rho_zeroand covariatesX_zeroCount part: a reduced-form SAR NB with parameter
rho, covariatesX, and dispersionalpha
We use a 50×50 grid (2500 observations) for a demo.
# Simulate data from the ZINB DGP
# Use n=50 (50x50 grid → 2500 observations) for reliable parameter recovery.
# Spatial parameters (rho, rho_zero) require n ≥ 900 for reliable recovery in
# cross-sectional count models. The model raises a UserWarning when n < 900.
data = simulate_sar_zinb(
n_side=50, # 50x50 grid → 2500 observations
rho=0.4, # Count SAR parameter
rho_zero=0.3, # Zero-part SAR parameter
alpha=5.0, # NB dispersion (larger = less overdispersion)
beta=np.array([1.0, 0.6]), # Count coefficients [intercept, slope]
beta_zero=np.array([0.3, 1.0]), # Zero-part coefficients [intercept, slope]
target_pi=0.7, # Target activation probability
seed=42,
latent=False,
)
y = data["y"]
X = data["X"]
X_zero = data["X_zero"]
W_graph = data["W_graph"]
W_zero_graph = data["W_zero_graph"]
print(f"Observations: {len(y)}")
print(f"Zero fraction: {(y == 0).mean():.2%}")
print(f"Mean (y>0): {y[y > 0].mean():.1f}")
print(f"Max: {y.max():.0f}")
print("\nTrue parameters:")
print(f" ρ (count SAR): {data['params_true']['rho']}")
print(f" ρ (zero SAR): {data['params_true']['rho_zero']}")
print(f" α (dispersion): {data['params_true']['alpha']}")
print(f" β (count coef): {data['params_true']['beta']}")
print(f" β (zero coef): {data['params_true']['beta_zero']}")
Observations: 2500
Zero fraction: 34.12%
Mean (y>0): 6.6
Max: 61
True parameters:
ρ (count SAR): 0.4
ρ (zero SAR): 0.3
α (dispersion): 5.0
β (count coef): [1. 0.6]
β (zero coef): [0.71603727 1. ]
Fit the model¶
The SARZINB model class accepts:
y: non-negative integer countsX: count covariate matrixX_zero: zero-part covariate matrix (defaults toXif not provided)W: spatial weights for the count partW_zero: spatial weights for the zero part (defaults toW)
The fit() method runs a 9-block Gibbs sampler with profile-log-likelihood initialisation.
# Create and fit the ZINB-SAR model
model = SARZINB(
y=y,
X=X,
X_zero=X_zero,
W=W_graph,
W_zero=W_zero_graph,
latent=False,
)
idata = model.fit(
draws=2000,
tune=1000,
chains=4,
random_seed=123,
n_jobs=-1,
progressbar=True,
)
Gibbs sampling (zinb_sar): 4 chains for 1,000 tune and 2,000 draw iterations (4 x 3,000 = 12,000 draws total)
/home/runner/micromamba/envs/test/lib/python3.14/site-packages/rich/live.py:260: UserWarning: install "ipywidgets"
for Jupyter support
warnings.warn('install "ipywidgets" for Jupyter support')
Sampling took 42s (285 draws/s)
Read the two equations separately¶
Compare posterior means to the true parameter values used in the DGP.
# Posterior summary
summary = az.summary(idata, var_names=["rho", "rho_zero", "alpha", "beta", "beta_zero"])
print(summary)
# Compare with true values
print("\n--- Parameter Recovery ---")
print(
f"ρ (count SAR): true={data['params_true']['rho']:.2f}, "
f"posterior mean={float(idata.posterior['rho'].mean()):.3f} "
f"± {float(idata.posterior['rho'].std()):.3f}"
)
print(
f"ρ (zero SAR): true={data['params_true']['rho_zero']:.2f}, "
f"posterior mean={float(idata.posterior['rho_zero'].mean()):.3f} "
f"± {float(idata.posterior['rho_zero'].std()):.3f}"
)
print(
f"α (dispersion): true={data['params_true']['alpha']:.2f}, "
f"posterior mean={float(idata.posterior['alpha'].mean()):.3f} "
f"± {float(idata.posterior['alpha'].std()):.3f}"
)
# Note: spatial parameters (rho, rho_zero) require large samples (n ≥ 900)
# for reliable recovery in cross-sectional count models. The posterior
# is typically wide and may be attenuated toward zero at moderate n.
# The model raises a UserWarning when n < 900.
mean sd eti89_lb eti89_ub ess_bulk ess_tail r_hat mcse_mean mcse_sd
rho 0.387 0.0405 0.32 0.45 6293 5841 1.00 0.00051 0.00043
rho_zero 0.214 0.089 0.067 0.35 4637 5208 1.00 0.0013 0.00095
alpha 5.06 0.372 4.5 5.7 4932 5879 1.00 0.0053 0.0038
beta[x0] 1.03 0.067 0.93 1.1 6232 5934 1.00 0.00085 0.00071
beta[x1] 0.5826 0.0179 0.55 0.61 5364 6698 1.00 0.00024 0.00015
beta_zero[z0] 0.792 0.102 0.64 0.96 4070 5062 1.00 0.0016 0.001
beta_zero[z1] 1.009 0.065 0.91 1.1 2228 3740 1.00 0.0014 0.00065
--- Parameter Recovery ---
ρ (count SAR): true=0.40, posterior mean=0.387 ± 0.040
ρ (zero SAR): true=0.30, posterior mean=0.214 ± 0.089
α (dispersion): true=5.00, posterior mean=5.057 ± 0.372
What to check before trusting the output¶
Check R-hat and effective sample sizes to verify MCMC convergence.
# Convergence diagnostics
az.plot_trace_dist(idata, var_names=["rho", "rho_zero", "alpha"], compact=False)
plt.tight_layout()
plt.show()
# R-hat and ESS
diagnostics = az.summary(idata, var_names=["rho", "rho_zero", "alpha"])
print(diagnostics[["r_hat", "ess_bulk", "ess_tail"]])
/tmp/ipykernel_6201/1222109080.py:3: UserWarning: The figure layout has changed to tight
plt.tight_layout()
r_hat ess_bulk ess_tail
rho 1.00 6293 5841
rho_zero 1.00 4637 5208
alpha 1.00 4932 5879
Split the zeros between the two processes¶
The ZINB model decomposes observed zeros into two types:
Structural zeros: the observation is inactive (d=0)
Sampling zeros: the observation is active (d=1) but the NB draw was zero
The zero_attribution() method computes the posterior probability of each type for every zero observation.
# Zero attribution
attribution = model.zero_attribution()
print(f"Number of zero observations: {len(attribution['zero_indices'])}")
print(f"\nMean P(structural | y=0): {attribution['structural_prob'].mean():.3f}")
print(f"Mean P(sampling | y=0): {attribution['sampling_prob'].mean():.3f}")
# Histogram of structural zero probabilities
fig, ax = plt.subplots(1, 2, figsize=(12, 4))
ax[0].hist(attribution["structural_prob"], bins=20, edgecolor="white")
ax[0].set_xlabel("P(structural zero | y=0)")
ax[0].set_ylabel("Count")
ax[0].set_title("Structural Zero Probabilities")
ax[1].hist(attribution["sampling_prob"], bins=20, edgecolor="white", color="C1")
ax[1].set_xlabel("P(sampling zero | y=0)")
ax[1].set_ylabel("Count")
ax[1].set_title("Sampling Zero Probabilities")
plt.tight_layout()
plt.show()
Number of zero observations: 853
Mean P(structural | y=0): 0.887
Mean P(sampling | y=0): 0.113
/tmp/ipykernel_6201/1143200135.py:19: UserWarning: The figure layout has changed to tight
plt.tight_layout()
Get per-observation activation probabilities¶
The activation_probabilities() method returns the posterior-mean probability that each observation is “active” (i.e., in the count regime rather than the always-zero regime).
# Activation probabilities
pi = model.activation_probabilities()
print(f"Mean activation probability:{pi.mean():.3f}")
print(f"Min activation probability: {pi.min():.3f}")
print(f"Max activation probability: {pi.max():.3f}")
# Compare with true d
d_true = data["d"]
print(f"\nTrue activation rate: {d_true.mean():.3f}")
print(f"Estimated activation rate: {pi.mean():.3f}")
# Scatter: pi vs observed y
fig, ax = plt.subplots(figsize=(8, 5))
ax.scatter(pi, y, alpha=0.5, s=20)
ax.set_xlabel("Estimated P(active)")
ax.set_ylabel("Observed count y")
ax.set_title("Activation Probability vs Observed Count")
ax.axhline(0, color="grey", linestyle="--", alpha=0.5)
plt.tight_layout()
plt.show()
Mean activation probability:0.698
Min activation probability: 0.084
Max activation probability: 0.990
True activation rate: 0.705
Estimated activation rate: 0.698
/tmp/ipykernel_6201/949050388.py:20: UserWarning: The figure layout has changed to tight
plt.tight_layout()
Predict expected counts¶
The fitted_values() method returns E[y_i] = π_i · exp(η_i^cnt), the posterior-mean expected count for each observation.
# Fitted mean counts
fitted = model.fitted_values()
print(f"Mean fitted count: {fitted.mean():.2f}")
print(f"Mean observed count: {y.mean():.2f}")
# Scatter: fitted vs observed
fig, ax = plt.subplots(figsize=(8, 5))
ax.scatter(fitted, y, alpha=0.5, s=20)
ax.plot(
[0, max(y.max(), fitted.max())],
[0, max(y.max(), fitted.max())],
"r--",
alpha=0.5,
label="45° line",
)
ax.set_xlabel("Fitted E[y]")
ax.set_ylabel("Observed y")
ax.set_title("Fitted vs Observed Counts")
ax.legend()
plt.tight_layout()
plt.show()
Mean fitted count: 4.37
Mean observed count: 4.33
/tmp/ipykernel_6201/2483833273.py:21: UserWarning: The figure layout has changed to tight
plt.tight_layout()
Inspect the posteriors¶
Visualize the posterior distributions of the spatial parameters rho and rho_zero, and the NB dispersion alpha, with true values marked.
# Posterior distributions with true values
truth = data["params_true"]
pc = az.plot_dist(
idata, var_names=["rho", "rho_zero", "alpha"], figure_kwargs={"figsize": (14, 4)}
)
az.add_lines(pc, {name: truth[name] for name in ("rho", "rho_zero", "alpha")})
plt.show()
# Note: spatial parameters (rho, rho_zero) require large samples (n ≥ 900)
# for reliable recovery in cross-sectional count models. The posterior
# is typically wide and may be attenuated toward zero at moderate n.
See also¶
How to estimate spatial negative binomial models — fit this first and compare before assuming zero inflation
How to estimate Poisson origin–destination flow models — the flow-data version of the same question
Supported Models —
SARZINBequations and arguments