How to run spatial block cross-validation

You need an honest out-of-sample comparison between spatial models, and az.loo will not give you one — leave-one-out assumes observations are exchangeable given the parameters, which spatial dependence breaks. spatial_kfold holds out spatially contiguous blocks instead, refits on the remainder, and scores the held-out fold under the full-data joint implied by the model [Roberts et al., 2017].

The scoring rule and its derivation are in Architecture.

!pip install git+https://github.com/ljwolf/geovalidate/ --no-deps
Collecting git+https://github.com/ljwolf/geovalidate/
  Cloning https://github.com/ljwolf/geovalidate/ to /tmp/pip-req-build-bexa7ylw
  Running command git clone --filter=blob:none --quiet https://github.com/ljwolf/geovalidate/ /tmp/pip-req-build-bexa7ylw
  Resolved https://github.com/ljwolf/geovalidate/ to commit 6106134e10c1578fed6ee114153e720082c4e861
  Installing build dependencies ... ?25l-
 \
 |
 /
 -
 done
?25h  Getting requirements to build wheel ... ?25ldone
?25h  Preparing metadata (pyproject.toml) ... ?25ldone
?25hBuilding wheels for collected packages: geovalidate
  Building wheel for geovalidate (pyproject.toml) ... ?25ldone
?25h  Created wheel for geovalidate: filename=geovalidate-0.1.0-py3-none-any.whl size=68551 sha256=00f9627ba65a0b5edd008b20e25794f370f1d0cc07e4173260f2abc8d2883cb4
  Stored in directory: /tmp/pip-ephem-wheel-cache-cqv6g2gw/wheels/07/e1/1c/505deea87953881f24f62e5ffe6dc847b905492d550844705a
Successfully built geovalidate
Installing collected packages: geovalidate
Successfully installed geovalidate-0.1.0

Set up data with a known answer

We simulate from a SAR data-generating process on a regular grid so the “correct” model is known in advance. A well-behaved CV criterion should prefer the SAR over the misspecified OLS, SLX, and SEM alternatives.

import numpy as np
import pandas as pd
from libpysal.graph import Graph

from neighbayes.dgp import simulate_sar
from neighbayes.diagnostics import spatial_kfold
from neighbayes.models import OLS, SAR, SEM, SLX

SEED = 0
GRID = 12  # 12 x 12 = 144 cells
RHO_TRUE = 0.6
BETA_TRUE = np.array([1.0, 2.0, -1.5])

gdf = simulate_sar(
    n_side=GRID,
    rho=RHO_TRUE,
    beta=BETA_TRUE,
    sigma=1.0,
    seed=SEED,
    create_gdf=True,
    geometry_type="polygon",
)

W = Graph.build_contiguity(gdf, rook=True).transform("r")
print(f"n = {len(gdf)}, true rho = {RHO_TRUE}, columns = {list(gdf.columns)}")
n = 144, true rho = 0.6, columns = ['y', 'X_0', 'X_1', 'X_2', 'geometry']
/home/runner/micromamba/envs/test/lib/python3.14/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

Fit the models you want to compare

We fit four candidates — only the SAR matches the DGP — using short MCMC runs sufficient for a pedagogical example.

FIT_KW = dict(draws=400, tune=400, chains=2, random_seed=SEED, progressbar=False)
FORMULA = "y ~ X_1 + X_2"

models = {
    "OLS": OLS(formula=FORMULA, data=gdf, W=W),
    "SLX": SLX(formula=FORMULA, data=gdf, W=W),
    "SAR": SAR(formula=FORMULA, data=gdf, W=W, logdet_method="eigenvalue"),
    "SEM": SEM(formula=FORMULA, data=gdf, W=W, logdet_method="eigenvalue"),
}
for name, m in models.items():
    m.fit(**FIT_KW)
    print(f"  fit {name}")
  fit OLS
  fit SLX
  fit SAR
  fit SEM
Initializing NUTS using jitter+adapt_diag...
Multiprocess sampling (2 chains in 2 jobs)
NUTS: [beta, sigma2]
Sampling 2 chains for 400 tune and 400 draw iterations (800 + 800 draws total) took 2 seconds.
We recommend running at least 4 chains for robust computation of convergence diagnostics
Initializing NUTS using jitter+adapt_diag...
Multiprocess sampling (2 chains in 2 jobs)
NUTS: [beta, sigma2]
Sampling 2 chains for 400 tune and 400 draw iterations (800 + 800 draws total) took 2 seconds.
We recommend running at least 4 chains for robust computation of convergence diagnostics

Score them out of sample

spatial_kfold takes the fold design as a splitter: any object whose split method yields (train_idx, test_idx) pairs, the scikit-learn convention. Here we cut the 12 × 12 grid into sixteen 3 × 3 blocks and let scikit-learn’s GroupKFold deal them into 5 folds, so each fold holds out whole blocks. spatial_kfold refits each model on each training fold and accumulates the held-out elpd. Higher elpd is better.

from sklearn.model_selection import GroupKFold

CV_KW = dict(draws=200, tune=200, chains=1, random_seed=SEED, progressbar=False)

row, col = np.divmod(np.arange(len(gdf)), GRID)
blocks = (row // 3) * (GRID // 3) + col // 3  # sixteen 3 x 3 blocks

results = {}
for name, m in models.items():
    results[name] = spatial_kfold(m, GroupKFold(n_splits=5), groups=blocks, **CV_KW)

summary = pd.DataFrame(
    {
        "elpd": [r.elpd for r in results.values()],
        "se": [r.se for r in results.values()],
        "n_folds": [r.n_folds for r in results.values()],
    },
    index=list(results.keys()),
).sort_values("elpd", ascending=False)
summary["delta_elpd"] = summary["elpd"] - summary["elpd"].max()
summary
elpd se n_folds delta_elpd
SAR -210.101287 0.673073 5 0.000000
SLX -234.447969 1.061533 5 -24.346682
SEM -235.535469 0.579822 5 -25.434182
OLS -281.858445 1.283355 5 -71.757158

With data simulated from a SAR DGP, spatial block CV should rank SAR on top, with the misspecified alternatives trailing behind. The delta_elpd column shows the elpd difference relative to the best model — values close to zero indicate near-equivalent predictive performance, while large negative values indicate worse out-of-block prediction.

Swap in a different fold design

Hand-built blocks are coarse. For a richer menu of spatial cross-validation designs — Hilbert space-filling curves, DGGS-cell stratification, clustered hold-outs, and exclusion-buffered leave-one-out — use the splitters in geovalidate. They take the geometry as split’s first argument, so pass it as geometry=.

Install with pip install geovalidate.

Below we re-run CV using HilbertKFold, which interleaves observations along a Hilbert curve so every fold covers the entire study area.

Each refit uses the training units’ weights, row-standardized again after the held-out units are dropped. When a fold scatters its held-out units, many training units lose neighbors, and that refit is biased: on a 10 × 10 rook grid with 12% of units held out in a checkerboard, a SAR’s posterior mean of ρ fell from 0.57 to 0.42. Contiguous blocks cost far less (0.55 with a 4 × 4 block). Prefer designs that hold out compact regions when comparing lag models.

from geovalidate import HilbertKFold

splitter = HilbertKFold(n_splits=5, random_state=SEED)

results_hilbert = {
    name: spatial_kfold(m, splitter, geometry=gdf.geometry, **CV_KW)
    for name, m in models.items()
}

summary_hilbert = pd.DataFrame(
    {
        "elpd": [r.elpd for r in results_hilbert.values()],
        "se": [r.se for r in results_hilbert.values()],
        "method": [r.method for r in results_hilbert.values()],
    },
    index=list(results_hilbert.keys()),
).sort_values("elpd", ascending=False)
summary_hilbert["delta_elpd"] = summary_hilbert["elpd"] - summary_hilbert["elpd"].max()
summary_hilbert
elpd se method delta_elpd
SAR -207.001774 0.892350 HilbertKFold 0.000000
SEM -226.617401 1.624752 HilbertKFold -19.615627
SLX -237.219069 1.038330 HilbertKFold -30.217294
OLS -280.125493 1.441062 HilbertKFold -73.123719

The method column records the splitter’s class name (HilbertKFold). Any other geovalidate splitter slots in the same way — for example LeaveClusterOut for leave-one-region-out CV, or BallKFold for spatially-exclusive folds with an explicit exclusion radius.

See also