[[8]{.chapter-number}  [Spatial Autocorrelation]{.chapter-title}]{#sec-esda .quarto-section-identifier}

8 Spatial Autocorrelation

Although it may be viewed as a special case of correlation, spatial autocorrelation has a meaning all its own. Whereas correlation statistics were designed to show relationships between variables, autocorrelation statistics are designed to show correlations within variables, and spatial autocorrelation shows the correlation within variables across space

– Getis (2007)

This chapter introduces several core components of exploratory spatial data analysis (ESDA), both conceptual and technical. Among the most important concepts in spatial analysis is the notion of spatial dependence (Anselin, 1988; Getis, 2008; Getis & Aldstadt, 2010), which (as its name suggests) codifies the idea that a value measured at some location (home price values, infection rates, job growth, etc.) is related (indeed, dependent) on the values nearby. This dependence (in the measured outcome) can result from measurement error, endogenous spatial spillover, or some exogenous spatially-correlated process. In this terminology, dependence is equivalent to causality, and we are reminded yet again that causality is rarely if ever provable. Thus we are often interested in examining the empirical association or correlation between measurements in space, and several such measures are well-developed in the spatial analysis literature. These statistics are useful for a wide variety of urban research questions, e.g.

does there appear to be some spatial process at play (i.e. is the spatial pattern of values random in space?)?
at what scale do spatial relationships appear to operate (i.e. do the same intuitions hold when varying the notion of “neighborhood” in the spatial graph)?
are there stronger patterns in some parts of the map (i.e. is the pattern driven by spatial clusters, hotspots, and coldspots)?

These are all useful research questions in isolation but can also be used inductively to generate additional hypotheses (hence the literature’s traditional emphasis on exploratory spatial data analysis). They can also be used as critical diagnostics at various stages of a research project, for example when considering whether spatial analysis is even necessary for some urban policy question, or examining whether model residuals are random in space–a critical but oft-unexamined econometric test (a topic explored in further depth in Chapter 28).

The focus on spatial association implies the existence of another core concept: the absence of any spatial association means there is no detectable pattern in the mapped values, which amounts to a state we call “complete spatial randomness” (CSR). The state of CSR defines the null hypothesis in our test statistics for spatial association; it represents the counterfactual situation we test against, asking, “is our measured statistic (of spatial association) ‘very different’ from the measure we obtain if the map pattern was guaranteed to be random?”. Classic statistical inference quantifies that “very different” bit using \(p\)-values, which conceptually represent the probability of observing the statistic under the null hypothesis, and conventional estimation techniques require strong distributional assumptions (i.e. the statistic follows a standard Normal).

In spatial analyses, these asymptotic assumptions work in some cases but not others, and therefore, modern spatial analysis makes heavy use of computational inference techniques, particularly conditional random permutations, which provide an avenue for simulating CSR processes. Using this approach, an observed test statistic is evaluated against an empirical distribution of statistics computed on thousands of realizations of a CSR version of the same dataset. This non-parametric approach allows the computation of pseudo-\(p\)-values representing the share of cases when the observed statistic exceeds a critical threshold. For example, if an observed statistic is larger than 95% of the measures calculated on CSR simulations, then it would be “significant” at the 5% level (and its pseudo-\(p\)-value would be less than 0.5).

Classic measures of spatial association such as Geary’s \(C\), Moran’s \(I\), and Getis-Ord \(G\) have all been derived formally with analytical solutions, however, studies using simulated data show that the conditional permutation approach is often preferable to the analytical approach because the distributional assumptions of the latter fail easily, and permutational inference, therefore, tends to be more accurate in rejecting a “true null”. Since probabilities are just long-run frequencies, the pseudo-\(p\) value from permutational inference represents the probability of observing a result at least this extreme, given the observed data. The only real assumption of the permutational approach is that the distribution of permuted results (whatever its shape or scale) accurately reflects the null hypothesis (and here, we are quite confident the permutations achieve that). More issues related to conditional permutations are explored in Chapter 18.

8.1 The Geography of Educational Achievement

In this chapter we explore the concept of spatial dependence using empirical measures of spatial association, namely the join count statistic Moran’s \(I\) (both the global and local versions) using district-level data on math scores from national standardized exams. This allows us to explore both the mechanics of the statistics, and the related concept of complete spatial randomness (CSR), and understand what its results mean from a conceptual perspective.

Code

import contextily as cx
import esda
import geopandas as gpd
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sbn

from geosnap import DataStore
from geosnap import io as gio
from libpysal.graph import Graph
from splot.esda import lisa_cluster, moran_scatterplot, plot_moran

OMP: Info #276: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.

For this exercise, we’ll use two datasets:

a set of polygons (school districts) from the National Center for Educaiton Statistics
the Standford Education Data Archive (SEDA), which provides standardized achievement data for math and reading for most schools and school districts

Code

datasets = DataStore()

# get district boundaries for CA
dists = gio.get_nces(datasets, dataset="school_districts")
dists = dists[dists.STATEFP == "06"]

# collect SEDA achievement data & merge with boundaries
seda = datasets.seda(level="geodist", pooling="poolsub", accept_eula=True)
seda["math_score"] = seda.gcs_mn_avg_mth_eb - seda.gradecenter
seda = seda[seda["subcat"] == "all"]

# merge boundaries with achievement data
dists = dists.merge(seda, left_on="GEOID", right_on="sedalea")
dists = dists.dropna(subset=["math_score"])

# project into a reasonable coordinate system
dists = dists.to_crs(dists.estimate_utm_crs())

# and display the first few rows
dists.head()

/Users/knaaptime/miniforge3/envs/urban_analysis/lib/python3.12/site-packages/geosnap/_data.py:255: UserWarning: Streaming data from SEDA archive at <https://exhibits.stanford.edu/data/catalog/db586ns4974>.
Use `geosnap.io.store_seda()` to store the data locally for better performance
  warn(msg)

	STATEFP	GEOID	NAME	LOGRADE	HIGRADE	MTFCC	SDTYP	FUNCSTAT	ALAND	...	gcs_mn_avg_rla_eb	gcs_mn_grd_rla_eb	gcs_mn_coh_rla_eb	gcs_mn_avg_mth_eb_se	gcs_mn_grd_mth_eb_se	gcs_mn_coh_mth_eb_se	gcs_mn_avg_rla_eb_se	gcs_mn_grd_rla_eb_se	gcs_mn_coh_rla_eb_se	math_score
0	06	0600006	Ross Valley Elementary School District	KG	08	G5400	None	E	32811470.0	...	7.688360	1.023519	0.030797	0.076026	0.046664	0.020630	0.066978	0.041239	0.018836	1.531024
1	06	0600012	Twin Ridges Elementary School District	KG	08	G5400	None	E	742887130.0	...	3.832322	NaN	0.085849	0.146142	0.079741	0.036295	0.142507	NaN	0.034202	-1.974303
2	06	0601420	Fortuna Elementary School District	PK	08	G5400	None	E	75335396.0	...	4.385826	1.128633	0.006952	0.075341	0.043931	0.020358	0.065095	0.039649	0.018260	-1.423502
3	06	0601427	Penn Valley Union Elementary School District	KG	08	G5400	None	E	214653897.0	...	4.984243	1.138207	0.085833	0.084125	0.048134	0.022522	0.075780	0.044452	0.020704	-0.771874
4	06	0601680	Ackerman Elementary School District	KG	08	G5400	None	E	11308883.0	...	6.459234	1.173573	0.146546	0.084395	0.047652	0.022841	0.079345	0.046055	0.021699	0.577499

5 rows × 57 columns

The values in the math_score variable are grade-cohort standardized math achievement scores for each school district in California estimated via Empirical Bayes. These are essentially z-scores, albeit standardized at the national scale. “Average achievement” in math, would therefore be a score of zero.

Code

dists.math_score.hist()

Note the original data is centered on the national average (at zero), so California is considerably behind.

Code

dists[["math_score", "NAME", "geometry"]].explore(
    "math_score",
    scheme="quantiles",
    cmap="RdBu_r",
    tiles="CartoDB Positron",
    style_kwds={"stroke": False},
)

Make this Notebook Trusted to load map: File -> Trust Notebook