neighbayes Replication of panelg_manual and sdemo_programs

This notebook reproduces representative examples from:

  • reference/panelg_manual.pdf (panel SDEM synthetic example, Chapter 2), and

  • reference/sdemo_programs (cross-sectional SDM and SDEM synthetic examples, Chapter 7).

It also performs explicit fidelity checks against data-generating-process (DGP) truth where available.

Scope and Faithfulness Criteria

Faithfulness in this notebook means:

  1. Matching the same DGP structure and parameter values used in the reference examples.

  2. Matching panel/cross-sectional structure and fixed-effects settings.

  3. Checking that posterior means recover true parameters/effects within reasonable Monte Carlo tolerance.

Note: sdemo_programs/chapter4 convex-combination estimators (with unknown gamma weights over multiple W matrices) are not currently implemented in neighbayes. This notebook uses examples that are exactly representable by current neighbayes classes.

import numpy as np
import pandas as pd
from libpysal.graph import Graph
from scipy.sparse import csr_matrix

from neighbayes import dgp
from neighbayes.models import SDEM, SDM
def make_knn_w(xcoord: np.ndarray, ycoord: np.ndarray, k: int) -> np.ndarray:
    """Build row-standardized k-NN weights (no self-neighbors)."""
    coords = np.column_stack([xcoord, ycoord])
    n = coords.shape[0]
    d = np.sqrt(((coords[:, None, :] - coords[None, :, :]) ** 2).sum(axis=2))
    np.fill_diagonal(d, np.inf)
    nn = np.argpartition(d, kth=k - 1, axis=1)[:, :k]

    W = np.zeros((n, n), dtype=float)
    for i in range(n):
        W[i, nn[i]] = 1.0

    rs = W.sum(axis=1, keepdims=True)
    rs[rs == 0.0] = 1.0
    return W / rs


def to_graph(W: np.ndarray) -> Graph:
    return Graph.from_sparse(csr_matrix(W)).transform("r")


def summarize_recovery(df: pd.DataFrame, abs_tol: float = 0.25) -> pd.DataFrame:
    out = df.copy()
    out["abs_error"] = (out["estimate"] - out["truth"]).abs()
    out["relative_error_pct"] = (
        100.0 * out["abs_error"] / np.maximum(np.abs(out["truth"]), 1e-12)
    )
    out["within_tol"] = out["abs_error"] <= abs_tol
    return out

Example A: panelg_manual Chapter 2 (sdem_panel_gd)

Reference DGP (manual pages around the SDEM section):

  • n = 100, t = 20, k = 2,

  • spatial error coefficient lambda = 0.7,

  • coefficients on X: +1, coefficients on WX: -1,

  • both region and time fixed effects (model = 3).

rng = np.random.default_rng(10203040)
n, t, k = 100, 20, 2
lam_true = 0.7
beta_true = np.ones(k)
theta_true = -np.ones(k)
sige = 0.1

x = rng.standard_normal((n * t, k))
Wn = make_knn_w(rng.random(n), rng.random(n), k=5)
Wbig = np.kron(np.eye(t), Wn)

SFE = np.kron(np.ones(t), np.arange(1, n + 1) / n)
TFE = np.kron(np.arange(1, t + 1) / t, np.ones(n))
u = np.linalg.solve(
    np.eye(n * t) - lam_true * Wbig, rng.standard_normal(n * t) * np.sqrt(sige)
)
y = x @ beta_true + (Wbig @ x) @ theta_true + SFE + TFE + u

m_panel = SDEM(
    unit_effects="fixed",
    time_effects=True,
    y=y,
    X=pd.DataFrame(x, columns=["x1", "x2"]),
    W=to_graph(Wn),
    N=n,
    T=t,
    priors={"lam_lower": -0.95, "lam_upper": 0.95},
)
idata_panel = m_panel.fit(
    draws=160, tune=160, chains=2, progressbar=False, random_seed=11
)
display(m_panel.summary(round_to=4))
mean sd eti89_lb eti89_ub ess_bulk ess_tail r_hat mcse_mean mcse_sd
lam 0.7141 0.0179 0.6832 0.7404 306.4221 241.5511 0.9987 0.0010 0.0008
sigma 0.3203 0.0053 0.3120 0.3295 282.2576 309.6130 0.9962 0.0003 0.0002
sigma2 0.1026 0.0034 0.0973 0.1086 282.2576 309.6130 0.9962 0.0002 0.0001
x1 1.0021 0.0082 0.9890 1.0151 400.3751 315.7780 1.0025 0.0004 0.0003
x2 1.0136 0.0087 1.0006 1.0277 329.2431 307.3361 0.9987 0.0005 0.0003
W*x1 -0.9899 0.0238 -1.0269 -0.9503 372.4332 283.6214 1.0004 0.0012 0.0009
W*x2 -0.9862 0.0247 -1.0251 -0.9466 295.9642 262.4585 0.9979 0.0014 0.0011
beta_hat = idata_panel.posterior["beta"].mean(("chain", "draw")).to_numpy()
lam_hat = float(idata_panel.posterior["lam"].mean(("chain", "draw")).to_numpy())
faith_panel = pd.DataFrame(
    {
        "term": ["lam", "x1", "x2", "W*x1", "W*x2"],
        "truth": [lam_true, 1.0, 1.0, -1.0, -1.0],
        "estimate": [lam_hat, beta_hat[0], beta_hat[1], beta_hat[2], beta_hat[3]],
    }
)
faith_panel = summarize_recovery(faith_panel, abs_tol=0.30)
display(faith_panel)
print("Panel SDEM pass rate:", faith_panel["within_tol"].mean())
term truth estimate abs_error relative_error_pct within_tol
0 lam 0.7 0.714109 0.014109 2.015531 True
1 x1 1.0 1.002088 0.002088 0.208798 True
2 x2 1.0 1.013623 0.013623 1.362327 True
3 W*x1 -1.0 -0.989870 0.010130 1.013023 True
4 W*x2 -1.0 -0.986245 0.013755 1.375513 True
Panel SDEM pass rate: 1.0

Example B: sdemo_programs Chapter 7 (sdm_cross_section_gd)

Reference DGP:

  • n = 1000, t = 1, rho = 0.5, beta = [1, 1], theta = [-0.5, -0.5],

  • intercept of 2,

  • W from k-NN coordinates with 6 neighbors.

rng = np.random.default_rng(30203040)
n, k = 1000, 2
rho_true = 0.5
intercept_true = 2.0
beta_true = np.ones(k)
theta_true = -0.5 * np.ones(k)

Wn = make_knn_w(rng.random(n), rng.random(n), k=6)
sdm_data = dgp.simulate_sdm(
    W=csr_matrix(Wn),
    rho=rho_true,
    beta1=np.array([intercept_true, *beta_true]),
    beta2=theta_true,
    sigma=np.sqrt(0.5),
    rng=rng,
)
y = sdm_data["y"]
X_arr = sdm_data["X"]

X = pd.DataFrame({"Intercept": X_arr[:, 0], "x1": X_arr[:, 1], "x2": X_arr[:, 2]})
m_sdm = SDM(
    y=y, X=X, W=sdm_data["W_graph"], priors={"rho_lower": -0.95, "rho_upper": 0.95}
)
idata_sdm = m_sdm.fit(draws=160, tune=160, chains=2, progressbar=False, random_seed=22)
display(m_sdm.summary(round_to=4))
/home/runner/micromamba/envs/test/lib/python3.14/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm
mean sd eti89_lb eti89_ub ess_bulk ess_tail r_hat mcse_mean mcse_sd
rho 0.4901 0.0378 0.4269 0.5464 340.6428 215.4459 1.0006 0.0021 0.0017
sigma 0.7026 0.0148 0.6784 0.7263 292.3323 287.3408 0.9977 0.0009 0.0007
sigma2 0.4939 0.0208 0.4602 0.5275 292.3323 287.3408 0.9977 0.0012 0.0009
Intercept 2.0631 0.1547 1.8375 2.3214 352.1266 279.0029 1.0002 0.0082 0.0069
x1 0.9870 0.0228 0.9495 1.0242 283.7723 260.6535 1.0083 0.0013 0.0009
x2 0.9889 0.0223 0.9542 1.0254 276.7285 321.2360 1.0012 0.0013 0.0009
W*x1 -0.5277 0.0653 -0.6358 -0.4233 328.1779 300.0760 1.0014 0.0036 0.0024
W*x2 -0.4564 0.0670 -0.5565 -0.3524 352.1096 264.5187 1.0183 0.0036 0.0043
post = idata_sdm.posterior
rho_hat = float(post["rho"].mean(("chain", "draw")).to_numpy())
beta_hat = post["beta"].mean(("chain", "draw")).to_numpy()

faith_sdm = pd.DataFrame(
    {
        "term": ["rho", "Intercept", "x1", "x2", "W*x1", "W*x2"],
        "truth": [
            rho_true,
            intercept_true,
            beta_true[0],
            beta_true[1],
            theta_true[0],
            theta_true[1],
        ],
        "estimate": [
            rho_hat,
            beta_hat[0],
            beta_hat[1],
            beta_hat[2],
            beta_hat[3],
            beta_hat[4],
        ],
    }
)

faith_sdm = summarize_recovery(faith_sdm, abs_tol=0.30)
display(faith_sdm)
print("Cross-sectional SDM fidelity pass rate:", faith_sdm["within_tol"].mean())
term truth estimate abs_error relative_error_pct within_tol
0 rho 0.5 0.490086 0.009914 1.982726 True
1 Intercept 2.0 2.063094 0.063094 3.154691 True
2 x1 1.0 0.987001 0.012999 1.299909 True
3 x2 1.0 0.988941 0.011059 1.105939 True
4 W*x1 -0.5 -0.527699 0.027699 5.539891 True
5 W*x2 -0.5 -0.456447 0.043553 8.710574 True
Cross-sectional SDM fidelity pass rate: 1.0

Example C: sdemo_programs Chapter 7 (sdem_cross_section_gd)

Reference DGP:

  • n = 3000, t = 1, rho = 0.8,

  • beta = [1,1,1,1], theta = 0.5 * beta,

  • intercept of 1,

  • SDEM disturbance process with k-NN(5) weights.

rng = np.random.default_rng(221010)
n, k = 3000, 4
lam_true = 0.8
intercept_true = 1.0
beta_true = np.ones(k)
theta_true = 0.5 * np.ones(k)

Wn = make_knn_w(rng.standard_normal(n), rng.standard_normal(n), k=5)
sdem_data = dgp.simulate_sdem(
    W=csr_matrix(Wn),
    lam=lam_true,
    beta1=np.array([intercept_true, *beta_true]),
    beta2=theta_true,
    sigma=1.0,
    rng=rng,
)
y = sdem_data["y"]
X_arr = sdem_data["X"]

X = pd.DataFrame({"Intercept": X_arr[:, 0]})
for j in range(k):
    X[f"x{j + 1}"] = X_arr[:, j + 1]

m_sdem = SDEM(
    y=y, X=X, W=sdem_data["W_graph"], priors={"lam_lower": -0.95, "lam_upper": 0.95}
)
idata_sdem = m_sdem.fit(
    draws=160, tune=160, chains=2, progressbar=False, random_seed=33
)
display(m_sdem.summary(round_to=4))
mean sd eti89_lb eti89_ub ess_bulk ess_tail r_hat mcse_mean mcse_sd
lam 0.8159 0.0095 0.8009 0.8299 244.0649 281.3278 1.0054 0.0006 0.0004
sigma 1.0018 0.0137 0.9811 1.0251 280.1189 223.5716 0.9967 0.0008 0.0005
sigma2 1.0038 0.0276 0.9625 1.0509 280.1189 223.5716 0.9967 0.0016 0.0010
Intercept 0.8752 0.1063 0.6978 1.0458 334.9328 240.5743 1.0009 0.0059 0.0048
x1 1.0461 0.0223 1.0102 1.0846 377.9078 378.0001 0.9967 0.0012 0.0008
x2 0.9753 0.0231 0.9399 1.0103 334.7009 207.0951 1.0051 0.0013 0.0011
x3 1.0207 0.0202 0.9883 1.0516 280.3655 255.6162 1.0033 0.0012 0.0007
x4 1.0251 0.0224 0.9925 1.0594 283.3348 269.1512 1.0017 0.0013 0.0010
W*x1 0.5498 0.0638 0.4467 0.6481 284.4825 309.9206 0.9961 0.0038 0.0022
W*x2 0.4788 0.0678 0.3755 0.5891 330.3044 310.2290 0.9996 0.0037 0.0025
W*x3 0.6112 0.0592 0.5114 0.7013 182.9535 195.0341 1.0080 0.0042 0.0023
W*x4 0.6313 0.0652 0.5246 0.7337 290.9596 244.2561 1.0057 0.0038 0.0026
post = idata_sdem.posterior
lam_hat = float(post["lam"].mean(("chain", "draw")).to_numpy())
beta_hat = post["beta"].mean(("chain", "draw")).to_numpy()

terms = (
    ["lam", "Intercept"]
    + [f"x{i + 1}" for i in range(k)]
    + [f"W*x{i + 1}" for i in range(k)]
)
truth_vals = [lam_true, intercept_true] + beta_true.tolist() + theta_true.tolist()
est_vals = (
    [lam_hat, beta_hat[0]]
    + beta_hat[1 : 1 + k].tolist()
    + beta_hat[1 + k : 1 + 2 * k].tolist()
)

faith_sdem = pd.DataFrame({"term": terms, "truth": truth_vals, "estimate": est_vals})
faith_sdem = summarize_recovery(faith_sdem, abs_tol=0.30)
display(faith_sdem)
print("Cross-sectional SDEM fidelity pass rate:", faith_sdem["within_tol"].mean())
term truth estimate abs_error relative_error_pct within_tol
0 lam 0.8 0.815895 0.015895 1.986924 True
1 Intercept 1.0 0.875157 0.124843 12.484257 True
2 x1 1.0 1.046083 0.046083 4.608276 True
3 x2 1.0 0.975335 0.024665 2.466458 True
4 x3 1.0 1.020677 0.020677 2.067729 True
5 x4 1.0 1.025055 0.025055 2.505503 True
6 W*x1 0.5 0.549798 0.049798 9.959503 True
7 W*x2 0.5 0.478798 0.021202 4.240400 True
8 W*x3 0.5 0.611212 0.111212 22.242454 True
9 W*x4 0.5 0.631268 0.131268 26.253514 True
Cross-sectional SDEM fidelity pass rate: 1.0

Overall Fidelity Verdict

This notebook demonstrates that neighbayes is faithful to the selected panel/cross-sectional examples from panelg_manual and sdemo_programs that are representable by the implemented model classes (SAR/SDM/SEM/SDEM/SLX with one W matrix and FE modes).

Caveat:

  • Convex-combination examples that estimate unknown gamma weights over multiple W matrices (for example *_conv_panel_g) are not currently implemented in neighbayes. Those are outside current fidelity scope.

Example D: semip-style SEM probit with random unit effects

This mirrors the core legacy semip_g structure:

  • binary outcome from latent threshold,

  • an effect per unit, each unit observed many times,

  • spatial dependence in the unit effects, a = lam * W * a + u.

Unlike legacy semip_g, this demonstration uses homoskedastic observation-level probit variance (no v_i/r hierarchy).

from neighbayes.models import SEMProbit

rng = np.random.default_rng(707)

# semip-like setup: m units with mobs observations each
m = 48
mobs = np.full(m, 60, dtype=int)
n = int(mobs.sum())
k = 3

# Unit-level kNN weights and spatial random effects
Wm = make_knn_w(rng.random(m), rng.random(m), k=6)
rho_true = 0.45
sigma_a_true = 1.2

u = rng.standard_normal(m) * sigma_a_true
a = np.linalg.solve(np.eye(m) - rho_true * Wm, u)

unit_ids = np.repeat(np.arange(m), mobs)
X = rng.standard_normal((n, k))
X = np.column_stack([np.ones(n), X])
feature_names = ["Intercept", "x1", "x2", "x3"]

beta_true = np.array([0.4, 0.8, -0.6, 0.5])
eta = X @ beta_true + a[unit_ids]
y = (eta + rng.standard_normal(n) > 0.0).astype(float)

sp = SEMProbit(
    y=y,
    X=pd.DataFrame(X, columns=feature_names),
    W=to_graph(Wm),
    units=unit_ids,
    unit_effects="random",
    priors={"lam_lower": -0.95, "lam_upper": 0.95, "beta_sigma": 5.0},
)
idata_sp = sp.fit(
    draws=220,
    tune=220,
    chains=1,
    target_accept=0.92,
    progressbar=False,
    random_seed=707,
)

display(sp.summary(var_names=["beta", "lam", "unit_effect_sd"], round_to=4))
Initializing NUTS using jitter+adapt_diag...
Sequential sampling (1 chains in 1 job)
NUTS: [lam, beta, unit_effect_sd, unit_effect_raw]
Sampling 1 chain for 220 tune and 220 draw iterations (220 + 220 draws total) took 5 seconds.
There were 4 divergences after tuning. Increase `target_accept` or reparameterize.
Only one chain was sampled, this makes it impossible to run some convergence checks
mean sd eti89_lb eti89_ub ess_bulk ess_tail r_hat mcse_mean mcse_sd
Intercept 0.8681 0.4419 0.3074 1.6144 17.2862 28.0494 NaN 0.1066 0.0421
x1 0.8669 0.0399 0.8068 0.9279 123.7409 165.7819 NaN 0.0036 0.0027
x2 -0.6227 0.0354 -0.6851 -0.5702 181.5795 163.2809 NaN 0.0027 0.0021
x3 0.5368 0.0383 0.4804 0.5948 131.2275 144.6861 NaN 0.0033 0.0031
lam 0.4165 0.1900 0.1130 0.6951 17.7372 17.7702 NaN 0.0450 0.0186
unit_effect_sd 1.2817 0.1317 1.0805 1.5097 54.8339 53.0328 NaN 0.0179 0.0094
beta_hat = idata_sp.posterior["beta"].mean(("chain", "draw")).to_numpy()
rho_hat = float(idata_sp.posterior["lam"].mean(("chain", "draw")).to_numpy())

a_mean_hat = sp.random_effects_mean().to_numpy()
a_rmse = float(np.sqrt(np.mean((a_mean_hat - a) ** 2)))

faith_sp = pd.DataFrame(
    {
        "term": ["lam"] + feature_names,
        "truth": [rho_true] + beta_true.tolist(),
        "estimate": [rho_hat] + beta_hat.tolist(),
    }
)
faith_sp = summarize_recovery(faith_sp, abs_tol=0.30)

print(f"Unit-effects RMSE: {a_rmse:.4f}")
display(faith_sp)
print("SEMProbit semip-style pass rate:", faith_sp["within_tol"].mean())
Unit-effects RMSE: 0.5292
SEMProbit semip-style pass rate: 0.8
term truth estimate abs_error relative_error_pct within_tol
0 lam 0.45 0.416457 0.033543 7.454053 True
1 Intercept 0.40 0.868138 0.468138 117.034518 False
2 x1 0.80 0.866854 0.066854 8.356773 True
3 x2 -0.60 -0.622694 0.022694 3.782339 True
4 x3 0.50 0.536795 0.036795 7.358923 True