How to run spatial block cross-validation¶
You need an honest out-of-sample comparison between spatial models, and
az.loo will not give you one — leave-one-out assumes observations are
exchangeable given the parameters, which spatial dependence breaks.
spatial_kfold holds out spatially contiguous blocks instead, refits on the
remainder, and scores the held-out fold under the full-data joint implied by the
model [Roberts et al., 2017].
The scoring rule and its derivation are in Architecture.
!pip install git+https://github.com/ljwolf/geovalidate/ --no-deps
Collecting git+https://github.com/ljwolf/geovalidate/
Cloning https://github.com/ljwolf/geovalidate/ to /tmp/pip-req-build-bexa7ylw
Running command git clone --filter=blob:none --quiet https://github.com/ljwolf/geovalidate/ /tmp/pip-req-build-bexa7ylw
Resolved https://github.com/ljwolf/geovalidate/ to commit 6106134e10c1578fed6ee114153e720082c4e861
Installing build dependencies ... ?25l-
\
|
/
-
done
?25h Getting requirements to build wheel ... ?25ldone
?25h Preparing metadata (pyproject.toml) ... ?25ldone
?25hBuilding wheels for collected packages: geovalidate
Building wheel for geovalidate (pyproject.toml) ... ?25ldone
?25h Created wheel for geovalidate: filename=geovalidate-0.1.0-py3-none-any.whl size=68551 sha256=00f9627ba65a0b5edd008b20e25794f370f1d0cc07e4173260f2abc8d2883cb4
Stored in directory: /tmp/pip-ephem-wheel-cache-cqv6g2gw/wheels/07/e1/1c/505deea87953881f24f62e5ffe6dc847b905492d550844705a
Successfully built geovalidate
Installing collected packages: geovalidate
Successfully installed geovalidate-0.1.0
Set up data with a known answer¶
We simulate from a SAR data-generating process on a regular grid so the “correct” model is known in advance. A well-behaved CV criterion should prefer the SAR over the misspecified OLS, SLX, and SEM alternatives.
import numpy as np
import pandas as pd
from libpysal.graph import Graph
from neighbayes.dgp import simulate_sar
from neighbayes.diagnostics import spatial_kfold
from neighbayes.models import OLS, SAR, SEM, SLX
SEED = 0
GRID = 12 # 12 x 12 = 144 cells
RHO_TRUE = 0.6
BETA_TRUE = np.array([1.0, 2.0, -1.5])
gdf = simulate_sar(
n_side=GRID,
rho=RHO_TRUE,
beta=BETA_TRUE,
sigma=1.0,
seed=SEED,
create_gdf=True,
geometry_type="polygon",
)
W = Graph.build_contiguity(gdf, rook=True).transform("r")
print(f"n = {len(gdf)}, true rho = {RHO_TRUE}, columns = {list(gdf.columns)}")
n = 144, true rho = 0.6, columns = ['y', 'X_0', 'X_1', 'X_2', 'geometry']
/home/runner/micromamba/envs/test/lib/python3.14/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
from .autonotebook import tqdm as notebook_tqdm
Fit the models you want to compare¶
We fit four candidates — only the SAR matches the DGP — using short MCMC runs sufficient for a pedagogical example.
FIT_KW = dict(draws=400, tune=400, chains=2, random_seed=SEED, progressbar=False)
FORMULA = "y ~ X_1 + X_2"
models = {
"OLS": OLS(formula=FORMULA, data=gdf, W=W),
"SLX": SLX(formula=FORMULA, data=gdf, W=W),
"SAR": SAR(formula=FORMULA, data=gdf, W=W, logdet_method="eigenvalue"),
"SEM": SEM(formula=FORMULA, data=gdf, W=W, logdet_method="eigenvalue"),
}
for name, m in models.items():
m.fit(**FIT_KW)
print(f" fit {name}")
fit OLS
fit SLX
fit SAR
fit SEM
Initializing NUTS using jitter+adapt_diag...
Multiprocess sampling (2 chains in 2 jobs)
NUTS: [beta, sigma2]
Sampling 2 chains for 400 tune and 400 draw iterations (800 + 800 draws total) took 2 seconds.
We recommend running at least 4 chains for robust computation of convergence diagnostics
Initializing NUTS using jitter+adapt_diag...
Multiprocess sampling (2 chains in 2 jobs)
NUTS: [beta, sigma2]
Sampling 2 chains for 400 tune and 400 draw iterations (800 + 800 draws total) took 2 seconds.
We recommend running at least 4 chains for robust computation of convergence diagnostics
Score them out of sample¶
spatial_kfold takes the fold design as a splitter: any object whose
split method yields (train_idx, test_idx) pairs, the scikit-learn
convention. Here we cut the 12 × 12 grid into sixteen 3 × 3 blocks and let
scikit-learn’s GroupKFold deal them into 5 folds, so each fold holds out whole
blocks. spatial_kfold refits each model on each training fold and accumulates
the held-out elpd. Higher elpd is better.
from sklearn.model_selection import GroupKFold
CV_KW = dict(draws=200, tune=200, chains=1, random_seed=SEED, progressbar=False)
row, col = np.divmod(np.arange(len(gdf)), GRID)
blocks = (row // 3) * (GRID // 3) + col // 3 # sixteen 3 x 3 blocks
results = {}
for name, m in models.items():
results[name] = spatial_kfold(m, GroupKFold(n_splits=5), groups=blocks, **CV_KW)
summary = pd.DataFrame(
{
"elpd": [r.elpd for r in results.values()],
"se": [r.se for r in results.values()],
"n_folds": [r.n_folds for r in results.values()],
},
index=list(results.keys()),
).sort_values("elpd", ascending=False)
summary["delta_elpd"] = summary["elpd"] - summary["elpd"].max()
summary
| elpd | se | n_folds | delta_elpd | |
|---|---|---|---|---|
| SAR | -210.101287 | 0.673073 | 5 | 0.000000 |
| SLX | -234.447969 | 1.061533 | 5 | -24.346682 |
| SEM | -235.535469 | 0.579822 | 5 | -25.434182 |
| OLS | -281.858445 | 1.283355 | 5 | -71.757158 |
With data simulated from a SAR DGP, spatial block CV should rank SAR on top, with the misspecified alternatives trailing behind. The delta_elpd column shows the elpd difference relative to the best model — values close to zero indicate near-equivalent predictive performance, while large negative values indicate worse out-of-block prediction.
Swap in a different fold design¶
Hand-built blocks are coarse. For a richer menu of spatial cross-validation
designs — Hilbert space-filling curves, DGGS-cell stratification, clustered
hold-outs, and exclusion-buffered leave-one-out — use the splitters in
geovalidate. They take the geometry
as split’s first argument, so pass it as geometry=.
Install with pip install geovalidate.
Below we re-run CV using HilbertKFold, which interleaves observations along a
Hilbert curve so every fold covers the entire study area.
Each refit uses the training units’ weights, row-standardized again after the held-out units are dropped. When a fold scatters its held-out units, many training units lose neighbors, and that refit is biased: on a 10 × 10 rook grid with 12% of units held out in a checkerboard, a SAR’s posterior mean of ρ fell from 0.57 to 0.42. Contiguous blocks cost far less (0.55 with a 4 × 4 block). Prefer designs that hold out compact regions when comparing lag models.
from geovalidate import HilbertKFold
splitter = HilbertKFold(n_splits=5, random_state=SEED)
results_hilbert = {
name: spatial_kfold(m, splitter, geometry=gdf.geometry, **CV_KW)
for name, m in models.items()
}
summary_hilbert = pd.DataFrame(
{
"elpd": [r.elpd for r in results_hilbert.values()],
"se": [r.se for r in results_hilbert.values()],
"method": [r.method for r in results_hilbert.values()],
},
index=list(results_hilbert.keys()),
).sort_values("elpd", ascending=False)
summary_hilbert["delta_elpd"] = summary_hilbert["elpd"] - summary_hilbert["elpd"].max()
summary_hilbert
| elpd | se | method | delta_elpd | |
|---|---|---|---|---|
| SAR | -207.001774 | 0.892350 | HilbertKFold | 0.000000 |
| SEM | -226.617401 | 1.624752 | HilbertKFold | -19.615627 |
| SLX | -237.219069 | 1.038330 | HilbertKFold | -30.217294 |
| OLS | -280.125493 | 1.441062 | HilbertKFold | -73.123719 |
The method column records the splitter’s class name (HilbertKFold). Any other geovalidate splitter slots in the same way — for example LeaveClusterOut for leave-one-region-out CV, or BallKFold for spatially-exclusive folds with an explicit exclusion radius.
See also¶
How to run Bayesian LM specification tests — the in-sample counterpart, cheaper and often enough
Architecture — the held-out scoring rule
Supported Models — the candidate models compared here