How to estimate panel flow models

Origin–destination flows observed over several periods — trade by year, commuting by quarter. Two classes, each unrestricted or separable and each made a panel by passing T: SARFlow for continuous flows, SARNegBinFlow for counts.

Two things differ from the cross-sectional case and both will bite if missed: you must pass T, and the Gaussian classes expect the outcome on the latent SAR scale, so a lognormal outcome goes in as log(y). Equations and constructor arguments are in Supported Models.

import contextlib
import io
import time

import arviz as az
import numpy as np
import pandas as pd

from neighbayes.dgp import simulate_sar_flow, simulate_sar_negbin_flow
from neighbayes.models import SARFlow, SARNegBinFlow

T = 3
RHO_D, RHO_O, RHO_W = 0.20, 0.15, 0.05
DGP = dict(n=12, T=T, seed=0, rho_d=RHO_D, rho_o=RHO_O, beta_d=[1.0], beta_o=[0.4])
FIT = dict(draws=800, tune=800, chains=4, random_seed=0, progressbar=False)


def fit_timed(model, **kw):
    start = time.perf_counter()
    with (
        contextlib.redirect_stdout(io.StringIO()),
        contextlib.redirect_stderr(io.StringIO()),
    ):
        idata = model.fit(**kw)
    return idata, time.perf_counter() - start


def report(label, idata, secs, params=("rho_d", "rho_o")):
    ess = az.ess(idata, var_names=list(params))
    rhat = az.rhat(idata, var_names=list(params))
    row = {"seconds": secs}
    for p in params:
        row[p] = float(idata.posterior[p].mean())
    row["min ess"] = float(min(float(ess[p]) for p in params))
    row["max rhat"] = float(max(float(rhat[p]) for p in params))
    return pd.Series(row, name=label)
/home/runner/micromamba/envs/test/lib/python3.14/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

Fit continuous flows

The DGP returns lognormal flows, so the model gets log(y). Without pair_effects or time_effects the panel has no effects; the flow panel classes take pair_effects where the other panel families take unit_effects.

data = simulate_sar_flow(rho_w=RHO_W, sigma=0.5, **DGP, pair_effects="fixed")
G, X, names = data["G"], data["X"], data["col_names"]

print(f"periods      : {T}")
print(f"flows/period : {len(data['y']) // T}")
print(f"true rho     : rho_d={RHO_D}, rho_o={RHO_O}, rho_w={RHO_W}")

model_free = SARFlow(y=np.log(data["y"]), W=G, X=X, T=T, col_names=names)
idata_free, secs_free = fit_timed(model_free, **FIT)

az.summary(idata_free, var_names=["rho_d", "rho_o", "rho_w", "sigma"]).round(3)
periods      : 3
flows/period : 144
true rho     : rho_d=0.2, rho_o=0.15, rho_w=0.05
mean sd eti89_lb eti89_ub ess_bulk ess_tail r_hat mcse_mean mcse_sd
rho_d 0.2 0.055 0.11 0.29 228 345 1.02 0.0036 0.0019
rho_o 0.238 0.058 0.14 0.33 196 340 1.02 0.0041 0.0025
rho_w 0.038 0.032 0.0041 0.098 674 539 1.01 0.0012 0.001
sigma 0.6968 0.0248 0.66 0.74 2902 3041 1.00 0.00046 0.00034

Look at ess_bulk and r_hat on the three ρ before anything else. They will be poor, and that is the model rather than the sampler: with \(\rho_d\), \(\rho_o\) and \(\rho_w\) all free they trade off along a nearly flat ridge that one-at-a-time updates cannot traverse.

Prefer the separable model

SARFlow(separable=True) pins \(\rho_w = -\rho_d\rho_o\), which removes the ridge outright. Same data, same draws.

data_sep = simulate_sar_flow(sigma=0.5, **DGP, separable=True, pair_effects="fixed")
model_sep = SARFlow(
    separable=True,
    y=np.log(data_sep["y"]),
    W=data_sep["G"],
    X=data_sep["X"],
    T=T,
    col_names=data_sep["col_names"],
)
idata_sep, secs_sep = fit_timed(model_sep, **FIT)

pd.DataFrame(
    [
        report("SARFlow panel (3 free rho)", idata_free, secs_free),
        report("SARFlow panel (separable)", idata_sep, secs_sep),
    ]
).round(3)
seconds rho_d rho_o min ess max rhat
SARFlow panel (3 free rho) 7.662 0.20 0.238 196.287 1.022
SARFlow panel (separable) 10.142 0.23 0.270 2070.214 1.000

The separable model buys well over an order of magnitude in effective samples for less time. Reach for it by default, and use the unrestricted model only when \(\rho_w\) is itself the object of interest — in which case expect to run far longer chains and to check convergence carefully.

You can confirm the restriction is exactly enforced rather than merely encouraged:

identity_error = float(
    np.max(
        np.abs(
            idata_sep.posterior["rho_w"].values
            + idata_sep.posterior["rho_d"].values * idata_sep.posterior["rho_o"].values
        )
    )
)
print(
    f"max |rho_w + rho_d*rho_o| = {identity_error:.2e}   (deterministic, not sampled)"
)
max |rho_w + rho_d*rho_o| = 0.00e+00   (deterministic, not sampled)

Fit flow counts

The negative binomial panel classes take integer counts directly — no log transform — and add a dispersion parameter alpha. They default to the reduced-form Gibbs sampler: the separable filter on its exact likelihood, the unrestricted one by Pólya–Gamma augmentation. With separable=True, pair_effects="fixed" adds a fixed effect per origin–destination pair, the fixed-effects panel identified by within-pair variation; the fits below have none.

nb = simulate_sar_negbin_flow(rho_w=RHO_W, alpha=2.0, **DGP)
nb_sep = simulate_sar_negbin_flow(alpha=2.0, **DGP, separable=True)

model_nb = SARNegBinFlow(
    y=nb["y"], W=nb["G"], X=nb["X"], T=T, col_names=nb["col_names"]
)
idata_nb, secs_nb = fit_timed(model_nb, **FIT)

model_nb_sep = SARNegBinFlow(
    separable=True,
    y=nb_sep["y"],
    W=nb_sep["G"],
    X=nb_sep["X"],
    T=T,
    col_names=nb_sep["col_names"],
)
idata_nb_sep, secs_nb_sep = fit_timed(model_nb_sep, **FIT)

pd.DataFrame(
    [
        report("SARNegBinFlow panel", idata_nb, secs_nb),
        report("SARNegBinFlow panel (separable)", idata_nb_sep, secs_nb_sep),
    ]
).round(3)
seconds rho_d rho_o min ess max rhat
SARNegBinFlow panel 38.920 0.127 0.132 1255.466 1.007
SARNegBinFlow panel (separable) 3.579 0.185 -0.277 948.240 1.003

What to check before trusting the output

r_hat and ess_bulk on every ρ, and — for the count models — on alpha too.

Warning

These panels are far too small to identify ρ The examples above use a 12-unit grid over 3 periods so the docs build in under a minute. At that size the spatial parameters of a flow model are not identified, and the point estimates below will not match the values the data were generated from — some come back with the wrong sign, and they move substantially from one random seed to the next while r_hat stays at 1.00 and ess_bulk runs into the thousands.

Good convergence diagnostics say the sampler explored the posterior faithfully. They say nothing about whether that posterior is concentrated near the truth. Read the credible intervals, notice how wide they are, and size your data accordingly — the count flow models want thousands of flows per period before ρ_d and ρ_o separate.

pd.DataFrame(
    [
        report("Gaussian, 3 free rho", idata_free, secs_free),
        report("Gaussian, separable", idata_sep, secs_sep),
        report("NegBin, 3 free rho", idata_nb, secs_nb),
        report("NegBin, separable", idata_nb_sep, secs_nb_sep),
    ]
).round(3)
seconds rho_d rho_o min ess max rhat
Gaussian, 3 free rho 7.662 0.200 0.238 196.287 1.022
Gaussian, separable 10.142 0.230 0.270 2070.214 1.000
NegBin, 3 free rho 38.920 0.127 0.132 1255.466 1.007
NegBin, separable 3.579 0.185 -0.277 948.240 1.003

See also