%load_ext watermark
%watermark -a 'eli knaap' -v -d -u -p geopandas,geosnapOMP: Info #276: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.Author: eli knaap
Last updated: 2025-02-09
Python implementation: CPython
Python version : 3.12.8
IPython version : 8.31.0
geopandas: 1.0.1
geosnap : 0.14.1.dev14+g0443e2a.d20250103
As a package focused on “neighborhoods”, much of the functionality in geosnap is organized around the concept of ‘endogenous’ neighborhoods. That is, it takes a classical perspective on neighborhood formation: a ‘neighborhood’ is defined loosely by its social composition, and the dynamics of residential mobility mean that these neighborhoods can grow, shrink, or transition entirely.
But two alternative concepts of “neighborhood” are also worth considering. The first, posited by social scientists, is that each person or household can be conceptialized as residing in its own neighborhood which extends outward from the person’s household until some threshold distance (Hipp & Boessen, 2013; Kim & Hipp, 2019). This neighborhood represents the boundary inside which we might expect some social interaction to occur with one’s ‘neighbors’ (Grannis, 1998, 2005). The second is a normative concept advocated by architects, urban designers, and planners (arguably still the goal for new urbanists): that a neighborhood is a discrete pocket of infrastructure organized as a small, self-contained economic unit (Perry, 1929). A common shorthand today is the 20 minute neighborhood.
The difference between these two perspectives is really what defines the origin of the neighborhood (the ‘town center’ or the person’s home), and whether ‘neighborhood’ is universal to all neighbors or unique for each resident. Both of them rely abstractly on the concept of an isochrone: the set of destinations accessible within a specified travel budget. Defining an isochrone is a network routing problem, but its spatial representation is usually given as a polygon that covers the set of locations. That polygon is also sometimes called a walkshed (or *travel shed, depending on the particular mode). For the person at the center of the isochrone, whose “neighborhood” the isochrone represents, the polygon is sometimes conceptualized as an “egohood” or “bespoke neighborhood”.
The trouble with generating isochrones is twofold. First, they are computationally intensive to create. You need to know the shortest path from an origin to all other possible destinations inside the travel threshold, and depending on the density of the network, this can scale masssively. Second, they are not straightforward to present cartographically/visually. The set of destinations reachable within the threshold is technically a set of points. If you’re constrained to a network, then we can usually assume that you also have access to potential locations between discrete destinations. For example if you’re walking along the sidewalk, you can stop at any house along the block until you reach your threshold distance. But sometimes that’s not the case. If you walk to a subway station, you can stop anywhere along the walk–until you get into the subway, then you are stuck traveling in one direction until you reach another station, then you get freedom of mobility again.
The latter case is particularly difficult to represent because it does not create a “valid” polygon… there’s a single polygon in the pre-transit zone, then the “accessible zone” collapses to a line (or nothing at all?) before opening back up into a polygon again. Like a barbell. If you take the simple route and just draw a convex hull around the accessible destinations, it will fail to collapse along the line, giving the false impression of much more ‘access’ than is realistic.
But then again, sometimes these off-network spaces are actually traversable. If two network nodes inside the travel threshold are separated by a park, then you can just walk through the park and that should be included in the visualization. If they are separated by a harbor or a mountain, you definitely can’t walk through (ok, maybe you could get through, but not without sacrificing a bit of speed at the very least).
To handle these issues, the isochrones implemented in geosnap take a straightforward approach, achieving a good balance between accuracy and speed. Specifically, we tackle the problem in two stages: first, we use pandana to generate the set nodes accessible from [a set of] destination[s]. Then we wrap an alpha shape around those destinations to create a tightly-fitted polygon. The alpha shape algorithm implemented in libpysal is also blazing fast, so the approach has worked quite well in all of our applications
from geosnap.analyze import isochrones_from_id, isochrones_from_gdf, pdna_to_adj
from geosnap.io import get_acs
from geosnap import DataStore
import pandana as pdna
import geopandas as gpd
import pandas as pd
import osmnx as ox
from geosnap import DataStore
from geosnap.io import get_acs, get_network_from_gdf
from geosnap import analyze as gaz
from geosnap import io as gio
from routingpy import Valhalla
from shapely.geometry import Polygon
datasets = DataStore()To generate a routable network, use the geosnap get_network_from_gdf function. Alternatively, you can also use pandana or urbanaccess directly. In a pinch, you can also download one of the metropolitan-scale pedestrian networks for the U.S. from geosnap’s quilt bucket, however these networks are several years old and intended primaryly for pedagogical purposes. The files are named for each CBSA fips code and extend 8km beyond the metro region’s borders to help mitigate edge effects. Here, we’ll use the quilt version from the San Diego region.
sd_tracts = get_acs(datasets, county_fips='06073', years=[2018])/Users/knaaptime/Dropbox/projects/geosnap/geosnap/io/constructors.py:217: UserWarning: Currency columns unavailable at this resolution; not adjusting for inflation
warn(import os
if not os.path.exists('41740.h5'):
import quilt3 as q3
b = q3.Bucket("s3://spatial-ucr")
b.fetch("osm/metro_networks_8k/41740.h5", "./41740.h5")
sd_network = pdna.Network.from_hdf5("41740.h5")Generating contraction hierarchies with 16 threads.
Setting CH node vector of size 332554
Setting CH edge vector of size 522484
Range graph removed 143094 edges of 1044968
. 10% . 20% . 30% . 40% . 50% . 60% . 70% . 80% . 90% . 100%To generate a travel isochrone, we have to specify an origin node on the network. For demonstration purposes, we can randomly select an origin from the network’s nodes_df dataframe (or choose a pre-selected example). To get the nodes for a specific set of origins, you can use pandana’s get_node_ids function
from random import samplerandom_origin = sample(sd_network.nodes_df.index.unique().tolist(),1)[0]random_origin2421449453example_origin = 1985327805Nike, er… this study says that the average person walks about a mile in 20 minutes, so we can define the 20-minute neighborhood for a given household as the 1-mile walkshed from that house. To simplify the computation a little, we say that each house “exists” at it’s nearest intersection
(this is the abstraction pandana typically uses to simplify the problem when using data from OpenStreetMap. There’s nothing prohibiting you from creating an even more realistic version with nodes for each residence, as long as you’re comfortable creating the Network)
adj = pdna_to_adj(sd_tracts.set_index('geoid'), sd_network, threshold=1600, )
adj/Users/knaaptime/Dropbox/projects/geosnap/geosnap/analyze/network.py:70: UserWarning: Geometry is in a geographic CRS. Results from 'centroid' are likely incorrect. Use 'GeoSeries.to_crs()' to re-project geometries to a projected CRS before this operation.
node_ids = network.get_node_ids(origins.centroid.x, origins.centroid.y).astype(int)| origin | destination | cost | |
|---|---|---|---|
| 0 | 060730001001 | 060730001001 | 0.000000 |
| 136 | 060730001001 | 060730001002 | 987.565979 |
| 197 | 060730001001 | 060730002011 | 1221.943970 |
| 240 | 060730001001 | 060730002022 | 1465.890991 |
| 280 | 060730001002 | 060730001002 | 0.000000 |
| ... | ... | ... | ... |
| 1036301 | 060730166072 | 060730166071 | 1596.958008 |
| 1036304 | 060730166073 | 060730166073 | 0.000000 |
| 1036489 | 060730166073 | 060730166071 | 932.747009 |
| 1036512 | 060730166073 | 060730166072 | 982.960022 |
| 1036608 | 060730166073 | 060730166101 | 1229.456055 |
11391 rows × 3 columns
iso = isochrones_from_id(example_origin, sd_network, threshold=1600 ) # network is expressed in meters/Users/knaaptime/Dropbox/projects/geosnap/geosnap/analyze/network.py:70: UserWarning: Geometry is in a geographic CRS. Results from 'centroid' are likely incorrect. Use 'GeoSeries.to_crs()' to re-project geometries to a projected CRS before this operation.
node_ids = network.get_node_ids(origins.centroid.x, origins.centroid.y).astype(int)iso.explore()We can also look at how the isochrone or bespoke neighborhood changes size and shape as we consider alternative travel thresholds. Because of the underlying network configuration, changing the threshold often results in some areas of the “neighborhood” changing more than others
iso_multiple = isochrones_from_id(example_origin, sd_network, threshold=[1600, 2400, 3200] )/Users/knaaptime/Dropbox/projects/geosnap/geosnap/analyze/network.py:70: UserWarning: Geometry is in a geographic CRS. Results from 'centroid' are likely incorrect. Use 'GeoSeries.to_crs()' to re-project geometries to a projected CRS before this operation.
node_ids = network.get_node_ids(origins.centroid.x, origins.centroid.y).astype(int)iso_multiple.explore('distance', cmap='Blues_r', style_kwds={'fillOpacity':0.5})In this example it’s easy to see how the road network topology makes it easier to travel in some directions more than others. Greenspace squeezes the western portion of the 1600m (20 min) isochrone into a horizontal pattern along Calle del Cerro, but the 2400m (30 minute) isochrone opens north-south tavel again along Avienda Pico, providing access to two other pockets of development, including San Clemente High School
We can also compare the network-based isochrone to the as-the-crow-flies approximation given by a euclidean buffer. If we didn’t have access to network data, this would be our best estimate of the shape and extent of the 20-minute neighborhood.
sd_network.nodes_df['geometry'] = gpd.points_from_xy(sd_network.nodes_df.x, sd_network.nodes_df.y)example_point = gpd.GeoDataFrame(sd_network.nodes_df.loc[example_origin]).T.set_geometry('geometry')
example_point = example_point.set_crs(4326)
planar_iso = example_point.to_crs(example_point.estimate_utm_crs()).buffer(1600)
m = planar_iso.to_crs(4326).explore()
iso.explore(m=m)Obviously from this depiction, network-constrained travel is very different from a euclidean approximation. That’s especially true in places with irregular networks or topography considerations (like much of California)
the isochrones function calculates several isochrones simultaneously, given a set of input destinations. For example we could look at the 20-minute neighborhood for schools in San Diego county.
from geosnap.io import get_ncessd = gio.get_acs(datasets, county_fips="06073", years=[2019], level="tract")
sd = sd[sd.n_total_pop > 0]
sd.plot()
schools = get_nces(datasets, dataset='schools')
sd_schools = schools[schools.to_crs(sd.crs).intersects(sd.union_all())]
sd_schools.head()| NCESSCH | NAME | OPSTFIPS | LSTREE | LCITY | LSTATE | LZIP | LZIP4 | STFIP15 | CNTY15 | ... | CBSATYPE15 | CSA15 | NMCSA15 | NECTA15 | NMNECTA15 | CD15 | SLDL15 | SLDU15 | geometry | year | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 5895 | 060004205341 | Warner Junior/Senior High | 06 | 30951 Highway 79 | Warner Springs | CA | 92086 | M | 06 | 06073 | ... | 1 | N | N | N | N | 0650 | 071 | 038 | POINT (-116.64292 33.27525) | 1516 |
| 5896 | 060004206527 | San Jose Valley Continuation Hig | 06 | 30951 Highway 79 | Warner Springs | CA | 92086 | M | 06 | 06073 | ... | 1 | N | N | N | N | 0650 | 071 | 038 | POINT (-116.64292 33.27525) | 1516 |
| 5897 | 060004206844 | Warner Elementary | 06 | 30951 Highway 79 | Warner Springs | CA | 92086 | 0008 | 06 | 06073 | ... | 1 | N | N | N | N | 0650 | 071 | 038 | POINT (-116.64292 33.27525) | 1516 |
| 5898 | 060004210387 | All Tribes Charter | 06 | 34320 Valley Center Rd. | Valley Center | CA | 92082 | 6046 | 06 | 06073 | ... | 1 | N | N | N | N | 0650 | 075 | 038 | POINT (-116.95367 33.27796) | 1516 |
| 5899 | 060004212735 | All Tribes Elementary Charter | 06 | 34320 Valley Center Rd. | Valley Center | CA | 92082 | 6046 | 06 | 06073 | ... | 1 | N | N | N | N | 0650 | 075 | 038 | POINT (-116.95367 33.27796) | 1516 |
5 rows × 26 columns
sd_schools.plot()
# randomly sample 25 schools and compute their walksheds
school_neighborhoods = isochrones_from_gdf(origins=sd_schools.sample(25), network=sd_network, threshold=1600,)/Users/knaaptime/Dropbox/projects/geosnap/geosnap/analyze/network.py:248: UserWarning: Geometry is in a geographic CRS. Results from 'centroid' are likely incorrect. Use 'GeoSeries.to_crs()' to re-project geometries to a projected CRS before this operation.
node_ids = network.get_node_ids(origins.centroid.x, origins.centroid.y).astype(int)
/Users/knaaptime/Dropbox/projects/geosnap/geosnap/analyze/network.py:70: UserWarning: Geometry is in a geographic CRS. Results from 'centroid' are likely incorrect. Use 'GeoSeries.to_crs()' to re-project geometries to a projected CRS before this operation.
node_ids = network.get_node_ids(origins.centroid.x, origins.centroid.y).astype(int)
/Users/knaaptime/Dropbox/projects/geosnap/geosnap/analyze/network.py:266: UserWarning: use_edges is True, but edge geometries are not available in the Network object. Try recreating with geosnap.io.get_network_from_gdf
warn(school_neighborhoods.head()| geometry | distance | |
|---|---|---|
| origin | ||
| 13720 | POLYGON ((-117.17202 32.80325, -117.17209 32.8... | 1600 |
| 8395 | POLYGON ((-117.26358 33.05799, -117.26474 33.0... | 1600 |
| 13574 | POLYGON ((-117.11238 32.71317, -117.11145 32.7... | 1600 |
| 13644 | POLYGON ((-117.07985 32.74375, -117.08051 32.7... | 1600 |
| 7216 | POLYGON ((-116.93544 32.80008, -116.93544 32.8... | 1600 |
school_neighborhoods.explore()If we adopt the “neighborhood unit” perspective, it might be reasonable to put a school at the center of the neighborhood, as it would provide equitable access to all residents. In that case, these are your neighborhoods
Here, I’ve setup an instance of the valhalla routing server on a cloud server. It’s actually pretty easy–everything is in docker, but you won’t be able to follow along unless you setup your own server and edit the URL below accordingly. Alternatively, you can try to use the public Valhalla server, but you may be rate-limited.
Often, you need routing, or travel time matrices, or isochrones for a place, but commercial or cloud-based solutions are not an option. For example if:
All of those cases require more control over the routing than web services can provide, which leaves two strong options for local computation.
# grab 10 observations
sdsub = sd.head(10)coords = list(
zip(sdsub.geometry.centroid.get_coordinates().x.tolist(), sdsub.geometry.centroid.y)
)/var/folders/j8/5bgcw6hs7cqcbbz48d6bsftw0000gp/T/ipykernel_9577/1309834641.py:2: UserWarning: Geometry is in a geographic CRS. Results from 'centroid' are likely incorrect. Use 'GeoSeries.to_crs()' to re-project geometries to a projected CRS before this operation.
zip(sdsub.geometry.centroid.get_coordinates().x.tolist(), sdsub.geometry.centroid.y)One is to setup a locally-hosted open-source routing engine and query that server instead of a commercial server (as you would do with a geocoding service). This is nice because they are highly performant, but the drawbacks are:
First we setup a connection to the Valhalla server (this one is running on a server in my research lab, but you can point this anywhere). Alternatively, you could try the public Valhalla server
# no trailing slash, api_key is not necessary
# you need to be connected to tailscale for this to work
client = Valhalla(base_url="http://cogs-research:8002", timeout=6000)# valhalla can do one origin at a time, so only pass the first
test_isos = client.isochrones(
coords[0],
"pedestrian",
intervals=[
2500,
2000,
1500,
1000,
],
interval_type="distance",
)# turn the json response into gdfs
def gdf_from_routingpy(isos):
polys = [Polygon(i['geometry']['coordinates']) for i in isos.raw['features']]
gdf = gpd.GeoDataFrame(geometry=polys, crs=4326)
return gdfgdf = gdf_from_routingpy(test_isos)
gdf["distance"] = [
2500,
2000,
1500,
1000,
]gdf.explore("distance", cmap="YlOrBr_r", style_kwds={"fill_opacity": 0.3})sdnet = get_network_from_gdf(sd)Generating contraction hierarchies with 16 threads.
Setting CH node vector of size 330134
Setting CH edge vector of size 879530
Range graph removed 905202 edges of 1759060
. 10% . 20% . 30% . 40% . 50% . 60% . 70% . 80% . 90% . 100%gisos = pd.concat(
[
gaz.isochrones_from_gdf(
gpd.GeoDataFrame(sdsub.iloc[0].to_frame().T),
threshold=t,
network=sdnet,
ratio=0.15,
)
for t in [2500, 2000, 1500, 1000]
]
)
gisos = gisos.set_crs(4326)gisos.explore("distance", cmap="YlOrBr_r", style_kwds={"fill_opacity":0.3}, name="geosnap")gdf.explore('distance', cmap='YlOrBr_r', style_kwds={'fill_opacity':0.3}, name='valhalla')The isochrones are pretty consistent, so i think the differences between the two are:
m = gdf.head(1).explore(name='valhalla')
gisos[gisos['distance']==2500].explore(m=m, name='geosnap', color='red')
from folium import LayerControl
LayerControl().add_to(m)
msd.shape[0]**2391876client.matrix?# pairwise matrix for all tract centroids in SD county
coords = list(zip(sd.geometry.centroid.get_coordinates().x.tolist(), sd.geometry.centroid.y))/var/folders/j8/5bgcw6hs7cqcbbz48d6bsftw0000gp/T/ipykernel_9577/2715920804.py:2: UserWarning: Geometry is in a geographic CRS. Results from 'centroid' are likely incorrect. Use 'GeoSeries.to_crs()' to re-project geometries to a projected CRS before this operation.
coords = list(zip(sd.geometry.centroid.get_coordinates().x.tolist(), sd.geometry.centroid.y))One nice thing about this implementation is that it’s indifferent to the structure of the input network. It could be pedestrian-based (which is the most common), but you could also use urbanaccess to create a multimodal network by combining OSM and GTFS data (in the next section). For this reason, the isochrone function provided here can be much more useful than commercially-provided web services like mapbox, first because all the computation happens locally (so no paywall, and no unwanted data collection) and second because the researcher has full control over how the network is structured. This allows you to do the kinds of analyses common in actual transportation modeling and planning–like adding a new transit line, a new lane in the freeway, or a new toll road, and see how the accessibility surface responds (and how the location choices of households and businesses will reallocate given this new geography).
Running the above functions on that multimodal network would give a good sense of “baseline” accessibility in a study region. An interested researcher could then make some changes to the GTFS network, say, by increasing bus frequency along a given corridor, then create another combined network and compare results. Who would benefit from such a change, and what might be the net costs? And thus, the seeds of open and reproducible scenario planning have been sown :)