---------------------------------------------------------------------- This is the API documentation for the starsim_data library. ---------------------------------------------------------------------- ## Dataclasses Data-holding classes Dataset(name: str, layout: str, tier: int = 0, levels: tuple = (0,), sources: tuple = (), license: str = '', n_locations: int = 0, description: str = '', loader: object = None, locations: tuple = (), provenance: dict = ) -> None What a dataset is: its shape, its provenance and where to get it. Attributes: name (str): the dataset name, as passed to `load(which=...)`. layout (str): how the values are stored; see `LAYOUTS`. tier (int): 0 if the dataset is bundled in the wheel, 1 if it is a store this package publishes and `fetch()` downloads, 2 if `download()` gets it from its original publisher (see `access` and `function` for the same in words). levels (tuple): the administrative levels covered; `(0,)` for national data. sources (tuple): `Source` records, oldest revision first. license (str): the upstream data license, e.g. `'CC BY 3.0 IGO'`. n_locations (int): how many locations the dataset covers. description (str): a one-line summary. loader (callable): if set, `load()` delegates to it instead of reading a bundled store -- the hook for registering a dataset this package does not ship. locations (tuple): for a store that is fetched, the locations it holds, so that they can be listed without downloading it. provenance (dict): for a store that is fetched, the provenance recorded in it. Source(store: str, group: str = '', revision: str = '', url: str = '', sha256: str = '') -> None Where one revision of a dataset physically lives. ## Functions Public functions load(location, which='prem', kind='all', revision=None, level=None) Load data for a location. Args: location (str): location name, ISO-2/ISO-3 code, official name or alias, e.g. `'Kenya'`, `'KE'`, `'KEN'`; case-, accent- and punctuation-insensitive. which (str): dataset -- the contact matrices (`'prem'`, the default, which is `'prem2021'` or, for the ten locations it lacks, `'prem2017'`, and can also be asked for as `'contacts'`; or `'prem2021'` itself), or a demographic series (`'births'`, `'deaths'`, `'migration'`, `'population'`, `'fertility'`). See `list_datasets()`; all of `'prem2017'` and `'mistry2021'` are fetched with `fetch()`, and annual series are downloaded with `download()`. kind (str): for contact-matrix datasets only, the contact setting -- one of `'all'`, `'home'`, `'work'`, `'school'`, `'community'`. Ignored otherwise. revision (str/int): for demographic series only, the UN WPP revision to read (default: the newest bundled). See `list_revisions()`. level (int/str): for the node table only, the administrative level to return (None returns every level). See `load_nodes()`, the readable spelling of `load(..., which='nodes', level=...)`. Returns: sciris.dataframe: a drop-in replacement for a pandas `DataFrame`, with source provenance in `.attrs['provenance']`. For contact matrices, a square `(n, n)` matrix whose rows and columns are indexed by the start age of each bin (e.g. `0, 5, ..., 75`). For demographic series, a tidy frame -- `births`/`migration` have columns `[Time, CBR]`/`[Time, CNMR]`; `deaths` has `[Time, Sex, AgeGrpStart, mx]`; `population` has `[Time, Sex, AgeGrp, Value]`; `fertility` has `[Time, AgeGrpStart, ASFR]`. Raises: ValueError: if `which`, `kind` or `revision` is not recognized. KeyError: if the location is unknown, or is absent from this dataset. Example: ```python import starsim_data as ssd M = ssd.load(location='kenya') # (16, 16) contact matrix d = ssd.load(location='KEN', which='deaths') # tidy mortality ``` fetch(location, which, kind='all', refresh=False, verbose=None, **kwargs) Fetch a store this package publishes but does not bundle, caching it for next time. Unlike `load()`, this touches the network on a cache miss -- deliberately, and under a different name, so that `load()`'s "never needs the internet" guarantee stays intact. What it downloads is a file published with this package's releases and checked against its recorded SHA-256, so it is the same for everyone on a version. Data that is not packaged at all, such as annual UN WPP series, is passed on to `download()`, with a warning that it comes from an external source. Args: location (str): location name, ISO-2/ISO-3 code, official name or alias, exactly as for `load()`. A US state is `'United States:Massachusetts'`; see `resolve()`. which (str): a contact source that is not bundled (`'prem2017'`, `'mistry2021'`, and `'mistry2021_us'` for the US states), or `'population_us'` (the population of each US state by single year of age). See `list_fetchable()`. kind (str): for a contact source only, the contact setting, as for `load()`. refresh (bool): re-download even if the file is already cached. verbose (bool): report progress (default: `options.verbose`). kwargs (dict): for data that is not packaged, passed to `download()` (`years`, `variant`, `revision`). Returns: sciris.dataframe: exactly the schema `load()` returns for the same layout; `population_us` has that of `population`, for the one year 2023 and `Sex` `'Both sexes'`. Provenance is in `.attrs['provenance']`. Raises: ValueError: if `which` is not fetchable, or an argument does not apply to it. KeyError: if the location is unknown, or is absent from this dataset. Example: ```python import starsim_data as ssd M = ssd.fetch('Japan', which='prem2017', kind='home') ``` download(location, which='deaths', years=None, variant='Medium', revision=None, refresh=False, verbose=None) Download UN WPP data that is not packaged, caching it for next time. Unlike `load()` and `fetch()`, this reaches the UN's own server on a cache miss, which this package does not control: it may be slow, or revise its files. The name says so at the call site; see `provenance()` for pinning what was used. Args: location (str): location name, ISO-2/ISO-3 code, official name or alias, exactly as for `load()`. WPP is national, so not a subnational unit. which (str): `'births'`, `'deaths'`, `'migration'`, `'population'`, `'fertility'` (the bundled series, at annual resolution), `'life_table'` (the complete abridged life table), or `'indicators'` (every WPP indicator). See `list_downloadable()`. years (int/tuple/list): a single year; a 2-tuple `(start, end)`, treated as an inclusive range; or an explicit list of years. None means all of 1950-2100, which for life tables means downloading both halves (~290 MB). variant (str): the projection variant, e.g. `'Medium'` (default), `'High'`, `'Low'`, `'Constant fertility'`. Life tables are Medium-only upstream. revision (str/int): the WPP release to download, e.g. `2024`. Defaults to the newest revision bundled with this package. refresh (bool): re-download even if the file is already cached. verbose (bool): report progress (default: `options.verbose`). Returns: sciris.dataframe: for the five bundled series, exactly the schema `load()` returns. For `life_table`, `[Time, Sex, AgeGrpStart, mx, qx, px, lx, dx, Lx, Sx, Tx, ex, ax]` with a glossary in `.attrs['columns']`. For `indicators`, `Time` plus every indicator column. Provenance is in `.attrs['provenance']`. Raises: ValueError: if `which`, `years`, `variant` or `revision` cannot be satisfied. KeyError: if the location is unknown. Example: ```python import starsim_data as ssd d = ssd.download('kenya', which='deaths', years=(2020, 2024)) lt = ssd.download('KEN', which='life_table', years=2020) ``` plot(location, which=None, year=None, level=1, fig_kw=None) Plot data for a location, as a quick check before it goes into a model. Args: location (str): location name, ISO-2/ISO-3 code, official name or alias. which (str/list): dataset(s) to plot. None (default) draws a summary figure -- age pyramid, crude birth rate, mortality curve and contact matrix -- skipping any the location lacks. year (int): the year to show for the single-year panels (age pyramid, mortality curve). Defaults to the most recent year that is not a projection. level (int/str): the administrative level, for `which='nodes'` only. fig_kw (dict): passed to `plt.subplots`, e.g. `dict(figsize=(12, 8))`. Returns: matplotlib.figure.Figure: the figure, so it can be saved or further adjusted. Example: ```python import starsim_data as ssd fig = ssd.plot('kenya') fig = ssd.plot('kenya', which='population') ``` choropleth(location, level=1, value='population', shapes=None, ax=None, cmap='viridis', fig_kw=None, **kwargs) Draw a filled map of an administrative value -- the geometry version of `plot(which='nodes')`. Requires the `[spatial]` extra, because it needs real polygons. If you only want to see roughly where the units are, `plot(location, which='nodes')` does that with the bundled centroids and no extra dependencies. Args: location (str): location name, ISO-2/ISO-3 code, official name or alias. level (int/str): administrative level to draw (default 1). value (str): the column to color by. `'population'` (default) downloads it from WorldPop; any other name must already be a column of `shapes`. shapes (GeoDataFrame): draw these instead of downloading, e.g. a frame you have already merged your own values onto. ax (matplotlib.axes.Axes): draw into this axes rather than making a figure. cmap (str): the colormap. fig_kw (dict): passed to `plt.subplots` when creating a figure. **kwargs: passed to `GeoDataFrame.plot`, e.g. `scheme=`, `vmin=`, `vmax=`. Returns: matplotlib.axes.Axes: the axes drawn into. Raises: ImportError: if the `[spatial]` extra is not installed. ValueError: if `value` is not available. Example: ```python import starsim_data as ssd ax = ssd.choropleth('kenya', level=1) ``` provenance(which=None, revision=None) Return where a dataset came from: source, license, citation and build details. Args: which (str): dataset name; if None, return a dict keyed by every dataset. revision (str/int): for demographic series, the UN WPP revision (default newest). Returns: sciris.objdict: with keys including `source`, `source_url`, `license`, `citation`, `built` and `builder`. Contents vary by source. Example: ```python import starsim_data as ssd ssd.provenance('prem2021').license # Returns 'GPL-3.0' ``` datasets(tier=None) Return every known dataset as `{name: Dataset}`. Args: tier (int): if given, only datasets in this tier (0 bundled, 1 fetched, 2 downloaded). Returns: sciris.objdict: name -> `Dataset`, sorted by name. dataset(name) Return the `Dataset` descriptor for `name`, which may be an alias (see `ALIASES`). Raises: ValueError: if the dataset is unknown; the message lists close matches. unregister_dataset(name) Remove a previously registered dataset. Bundled datasets cannot be removed. get_layout(name) Return the shaper registered for a layout. Raises: ValueError: if the layout is unknown. data_dir() Return the directory holding the bundled data files. The single place the bundled data is located, so that a test (or a rebuild) can point the whole package at a different directory by patching this one function -- followed by `refresh()`, since discovery is cached. refresh() Forget the discovered datasets, so the stores are re-read on next use. Only needed if the bundled data changes within a running process, e.g. after a rebuild in a test. load_nodes(location, level=0) Load the node table for a location: centroids, areas, p-codes and hierarchy. Args: location (str): location name, ISO-2/ISO-3 code, official name or alias. level (int/str): administrative level -- 0 for the country itself, 1 for its first-level subdivisions. Accepts `'admin1'` / `'ADM1'` / `1`; see `parse_level()`. None returns every level at once. Returns: sciris.dataframe: one row per administrative unit, with columns `nodeid`, `location`, `iso3`, `level`, `name`, `pcode`, `dot_name`, `parent`, `lat`, `lon`, `area_km2`, `population` and `source`. Provenance is in `.attrs['provenance']`. Raises: KeyError: if the location is unknown, or has no boundaries in the node table. ValueError: if the level is not available. Example: ```python import starsim_data as ssd ssd.load_nodes('kenya', level=1).shape[0] # Returns 47 ``` distances(location=None, level=1, nodes=None, units='km', index='dot_name') Great-circle distances between node centroids -- the input a gravity model needs. The gravity formulation itself (`c_ij = k * P_i^a * P_j^b / D_ij^c`) belongs in the model, not in a data package: this provides `D_ij`, and `load_nodes()` provides `P_i`. Distances are great-circle (haversine) on a spherical Earth, which is within about 0.5% of the geodesic distance -- far inside the uncertainty of treating an administrative unit as a point in the first place. Args: location (str): location name, ISO-2/ISO-3 code, official name or alias. level (int/str): administrative level to use (default 1, the subdivisions). nodes (dataframe): use this node table instead of loading one, e.g. one from `download_nodes()` with population filled in, or a subset. units (str): `'km'` (default), `'m'`, or `'mi'`. index (str): which column labels the rows and columns -- `'dot_name'` (default), `'pcode'`, `'name'` or `'nodeid'`. Returns: sciris.dataframe: a square, symmetric matrix with a zero diagonal. Raises: ValueError: if `units` or `index` is not recognized, or there are no nodes. Example: ```python import starsim_data as ssd d = ssd.distances('kenya', level=1) d.loc['Kenya:Nairobi', 'Kenya:Mombasa'] # Returns 433.1 (km) ``` adjacency(location=None, level=1, nodes=None, index='dot_name') Which administrative units share a border: a square boolean matrix. Worked out from the polygons when the node table is built (units within about 100 m of each other count as touching, which absorbs slivers between neighboring polygons), so it needs no geospatial libraries and no download. Args: location (str): location name, ISO-2/ISO-3 code, official name or alias. level (int/str): administrative level (default 1; level 1 is the finest bundled). nodes (dataframe): label the rows and columns from this node table instead of loading one, e.g. one from `download_nodes()` with population filled in; it must be the units of `location` at `level`, in `nodeid` order or a subset. A table from `download_nodes()` at a level that is not bundled carries its own borders, so any level works that way. index (str): which column labels the rows and columns -- `'dot_name'` (default), `'pcode'`, `'name'` or `'nodeid'`. Returns: sciris.dataframe: a square, symmetric boolean matrix with a false diagonal. Raises: ValueError: if the level has no bundled borders, or `index` is not recognized. Example: ```python import starsim_data as ssd a = ssd.adjacency('kenya', level=1) a.loc['Kenya:Nairobi', 'Kenya:Kiambu'] # Returns True: they share a border ``` node_name(name) Render an administrative unit's name in this package's canonical style. `New York` -> `New_York`, `Ile-de-France` -> `Ile_de_France`. The same accent- and punctuation-folding as `locations.normalize`, but case is preserved: a `dot_name` is read by people as well as matched by machines. Shared by the node-table build and by `download_shapes()`, so that a shape's `dot_name` is always the same string as its node's. download_shapes(location, level=1, source=None, refresh=False, verbose=None) Download administrative boundaries with full geometry from their publisher, and cache them. Requires the `[spatial]` extra. For centroids, areas and populations without any geospatial dependency, use `load_nodes()` instead -- it is bundled and instant. Args: location (str): location name, ISO-2/ISO-3 code, official name or alias. level (int/str): administrative level; 0 is the country, 1 its first-level subdivisions. Accepts `'admin2'` / `'ADM2'` / `2`. source (str): `'unocha'`, `'geoboundaries'` or `'gadm'`. None (default) tries UNOCHA then geoBoundaries. GADM is never tried automatically -- it is licensed for non-commercial use only, so it must be asked for by name. refresh (bool): re-download even if the file is already cached. verbose (bool): report progress (default: `options.verbose`). Returns: geopandas.GeoDataFrame: columns `nodeid`, `location`, `iso3`, `level`, `name`, `pcode`, `dot_name`, `source`, `geometry`. `pcode` and `dot_name` match `load_nodes()` for the same source, so the two join directly. Raises: ImportError: if the `[spatial]` extra is not installed. ValueError: if the source or level is not recognized. KeyError: if the location is unknown, or no source has it at this level. Example: ```python import starsim_data as ssd gdf = ssd.download_shapes('kenya', level=1) ``` list_shape_sources() Return the boundary sources as a dataframe: coverage, levels and license. Returns: sciris.dataframe: with columns `[source, max_level, license, redistributable, universal, note]`. Example: ```python import starsim_data as ssd ssd.list_shape_sources() # One row per source; GADM's has redistributable=False ``` download_nodes(location, level=1, year=None, source=None, refresh=False, verbose=None) The node table with `population` filled in, from WorldPop, at any level and year. `load_nodes()` is instant and offline, with populations for 2020 at levels 0 and 1. This downloads the boundaries and the population raster for a country, sums the raster inside each unit, and returns the same table with the numbers filled in. At a bundled level, the boundaries are downloaded from whichever source the bundled node table used for this country, so the p-codes line up and the join is exact rather than by name. At a finer level (2 and up), the table is built from the boundaries themselves, as the bundled one is, and which units share a border is worked out too and kept in `.attrs['borders']`, for `adjacency(nodes=...)`. Args: location (str): location name, ISO-2/ISO-3 code, official name or alias. level (int/str): administrative level (default 1). year (int): population year (default: the node table's, so level 0 and level 1 are for the same year). source (str): override the boundary source; see `list_shape_sources()`. Leaving this alone is usually right -- see above. refresh (bool): re-download even if the files are already cached. verbose (bool): report progress (default: `options.verbose`). Returns: sciris.dataframe: the node table for this location and level, with `population` filled and `population_source` / `population_year` recorded in `.attrs['provenance']`. Raises: ImportError: if the `[spatial]` extra is not installed. KeyError: if the location, its boundaries, or its raster cannot be found. Example: ```python import starsim_data as ssd nodes = ssd.download_nodes('kenya', level=1) ``` download_raster(location, year=None, refresh=False, verbose=None) Download a WorldPop 1 km gridded-population raster for a country, and cache it. Args: location (str): location name, ISO-2/ISO-3 code, official name or alias. year (int): the year to download (2000-2030). Defaults to the year the bundled node table uses for national population, so the two are comparable. refresh (bool): re-download even if the file is already cached. verbose (bool): report progress (default: `options.verbose`). Returns: pathlib.Path: the cached GeoTIFF. Raises: ValueError: if the year is outside WorldPop's range. KeyError: if WorldPop has no raster for this country and year. Example: ```python import starsim_data as ssd ssd.download_raster('kenya', year=2020) ``` aggregate_raster(raster, shapes, by='pcode', nearest=None) Sum a raster inside each polygon: zonal statistics, keyed by an attribute. Args: raster (str/Path): a GeoTIFF, e.g. from `download_raster()`. shapes (GeoDataFrame): polygons, e.g. from `download_shapes()`. by (str): the column of `shapes` to key the result on. nearest (float): if given, also count a cell whose center is in no polygon but within this distance (in degrees) of one, for the nearest: boundaries are simplified at the coast, so otherwise a coastal city or a small island loses its people. Returns: dict: `{key: total}` for each polygon. Raises: ImportError: if the `[spatial]` extra is not installed. ValueError: if `by` is not a column of `shapes`. Example: ```python import starsim_data as ssd gdf = ssd.download_shapes('kenya', level=1) ssd.aggregate_raster(ssd.download_raster('kenya'), gdf) ``` rescale_to_national(nodes, location=None, year=None) Scale node populations so they sum to this package's national WPP total. WorldPop and UN WPP disagree by a percent or two, and which one is "right" depends on the question. This makes the choice explicit rather than leaving two national totals silently in play. Args: nodes (dataframe): a node table with `population` filled, from `download_nodes()`. location (str): the location; taken from the table if not given. year (int): the WPP year to match; taken from the table's provenance if not given. Returns: sciris.dataframe: a copy with `population` rescaled. Example: ```python import starsim_data as ssd nodes = ssd.rescale_to_national(ssd.download_nodes('kenya')) ``` list_datasets(detailed=False, tier=None) Return the available datasets: contact sources plus WPP demographic series. Args: detailed (bool/str): if True, return a dataframe with each dataset's name, description, access (bundled, fetched or downloaded) and location count, rather than just the names; if 'full', also its layout, admin levels, revisions and license. tier (int): restrict to one tier (0 bundled, 1 fetched, 2 downloaded). Returns: list or sciris.dataframe: dataset names, or a table of them. Example: ```python import starsim_data as ssd 'prem2021' in ssd.list_datasets() # Returns True ssd.list_datasets(detailed=True) # A summary table ssd.list_datasets(detailed='full') # Every column of the registry ``` list_locations(which, revision=None) Return the sorted locations available in dataset `which`. A fetched dataset lists its locations from the manifest, without downloading it. Example: ```python import starsim_data as ssd 'Kenya' in ssd.list_locations('prem2021') # Returns True ``` list_revisions() Return the bundled UN WPP revisions, oldest first; the last is the default. Example: ```python import starsim_data as ssd ssd.list_revisions() # Returns ['2024'] ``` list_fetchable() Return the datasets `fetch()` can retrieve: the stores published with this package. Example: ```python import starsim_data as ssd 'mistry2021' in ssd.list_fetchable() # Returns True ``` list_downloadable() Return the datasets `download()` can retrieve. Example: ```python import starsim_data as ssd 'life_table' in ssd.list_downloadable() # Returns True ``` resolve(location, valid=None, dataset=None) Resolve a location string to its canonical key. Args: location (str): location name, ISO-2/ISO-3 code, official name, or alias; case-, accent- and punctuation-insensitive. A subnational unit is `':'`, e.g. `'United States:Massachusetts'`. valid (collection): if given, the resolved key must be one of these (used to scope resolution to the locations actually present in a dataset). dataset (str): name of the dataset `valid` came from, for error messages only. Returns: str: the canonical location key, e.g. `'United_States'` or `'United_States:Massachusetts'`. Raises: KeyError: if the location is unknown, or is known but absent from `valid`. The message lists close matches where there are any. location_info(location) Return the metadata row for a location, as a `sciris.objdict`. Args: location (str): location name, ISO-2/ISO-3 code, or alias. Returns: sciris.objdict: with keys `location`, `level`, `iso_code`, `iso3`, `official_name`, `aliases`, `primary_contact_source`, `contact_sources`, `population_source`. Example: ```python import starsim_data as ssd ssd.location_info('USA').iso3 # Returns 'USA' ``` list_all_locations() Return every canonical location name known to the package, sorted. This is the full location table; use `list_locations(which)` for the subset actually present in a given dataset. parse_level(level) Parse an administrative level into an integer. Accepts `'admin1'`, `'ADM2'`, `'3'` or a bare `int`. Level 0 is the country; higher levels are progressively finer subdivisions (1 = state/province, 2 = district/county). The upper bound is 5, the finest level any of the boundary sources in `starsim_data.shapes` publishes. Args: level (str/int): the level to parse. Returns: int: the level number, 0 to 5. Raises: ValueError: if `level` cannot be parsed or is out of range. Example: ```python import starsim_data as ssd ssd.parse_level('ADM2') # Returns 2 ``` register_dataset(dataset, overwrite=False) Add a dataset to the registry. The hook for data this package does not ship: give the descriptor a `loader`, and `load(which=)` will call it as `loader(location, **kwargs)`. Args: dataset (Dataset): the descriptor to register. overwrite (bool): whether to replace an existing dataset of the same name. Example: ```python import starsim_data as ssd ds = ssd.registry.Dataset(name='mydata', layout='crude', tier=2, loader=lambda location, **kw: location) ssd.registry.register_dataset(ds) ssd.load('Kenya', which='mydata') # Returns 'Kenya' ssd.registry.unregister_dataset('mydata') ``` register_layout(name, shaper, overwrite=False) Register a function that turns a stored array into a dataframe. Args: name (str): the layout name, matching `spec['layout']` in the store. shaper (callable): called as `shaper(group, key, spec, opts)` and returning a `sciris.dataframe`. `opts` carries the per-call arguments that apply to some layouts but not others -- `kind` for contact matrices, `level` for nodes. overwrite (bool): whether to replace an existing layout of the same name. cache_dir(subdir=None, create=False) Return the directory where downloaded data is cached. Resolved at call time from `ssd.options.cache_dir`, falling back to a platform default (`~/.cache/starsim_data` on Linux, `~/Library/Caches/starsim_data` on macOS, `%LOCALAPPDATA%\starsim_data\Cache` on Windows). Nothing is created unless `create=True`, so merely asking where the cache is has no side effects. Args: subdir (str): optional subdirectory within the cache, e.g. `'wpp'`. create (bool): whether to create the directory if it does not exist. Returns: pathlib.Path: the cache directory. Example: ```python import starsim_data as ssd ssd.cache_dir().name # Returns 'starsim_data' ``` cache_info() Return a dataframe of what is currently cached: file, size, source URL and date. Returns: sciris.dataframe: with columns `[file, size, bytes, url, downloaded]`, empty if nothing is cached. Example: ```python import starsim_data as ssd ssd.cache_info() ``` cache_provenance(path) Return the recorded provenance for a cached file, or None if there is none. cached_download(url, filename=None, subdir=None, refresh=False, verbose=None) Download `url` into the cache, or return the existing path if already cached. Args: url (str): the URL to fetch. filename (str): name to save as (default: the last path component of the URL). subdir (str): subdirectory of the cache to put it in, e.g. `'wpp'`. refresh (bool): re-download even if a cached copy exists. verbose (bool): report progress (default: `ssd.options.verbose`). Returns: pathlib.Path: the path to the cached file. clear_cache(pattern=None, verbose=None) Delete cached downloads. Bundled data is never touched. Args: pattern (str): a glob matched against each cached file's path relative to the cache root, e.g. `'wpp/*'`. If None, the whole cache is removed. verbose (bool): report what was deleted (default: `ssd.options.verbose`). Returns: list: the paths that were deleted. Example: ```python import starsim_data as ssd ssd.clear_cache('wpp/*') ``` ## Constants Module-level constants and data options(*args, **kwargs) Settings for `starsim_data`; the instance to use is `starsim_data.options`. Follows Starsim's `ss.options` conventions: an `objdict` whose entries can be set by attribute, by call, or from an environment variable. Example: ```python import starsim_data as ssd ssd.options(verbose=False) ssd.options.reset() ``` ---------------------------------------------------------------------- This is the User Guide documentation for the package. ---------------------------------------------------------------------- ## Getting started ### Home {{< include ../../../README.md >}} ### Getting started `starsim_data` gives you national demographic data, age-structured contact matrices and subnational administrative units, ready to use in a disease model. This page walks through the main functions in the order you are likely to need them. None of the examples need an internet connection (though we have to admit, it seems pretty likely you have one). ## Installation ::: {.panel-tabset} ## pip ```bash pip install starsim_data ``` ## with spatial support ```bash pip install "starsim_data[spatial]" ``` ## from a clone ```bash git clone https://github.com/starsimhub/starsim_data pip install -e starsim_data ``` ::: The package includes about 9 MB of data. The `[spatial]` option also installs `geopandas`, `pyogrio` and `rastertoolkit`, which you only need for the boundary shapes and population maps described in [Spatial data](spatial.qmd). ## What's available ```{python} import starsim_data as ssd ssd.list_datasets(detailed=True) ``` `ssd.list_datasets()` on its own gives just the names, and `ssd.list_datasets(detailed='full')` adds each dataset's layout, administrative levels, revisions and license. The `access` column says how you get each dataset. **Bundled** data is inside the package and loads instantly, with no internet connection. **Fetched** data is prepared by this package but kept out of it to save space; it is downloaded from this package's own releases the first time you use it. **Downloaded** data comes from its original publisher, such as the UN. Both fetched and downloaded data are saved on your computer after the first use. [Bundled, fetched and downloaded data](access.qmd) explains the difference. ## Loading a dataset ```{python} deaths = ssd.load('kenya', which='deaths') deaths.head() ``` The result is a `sciris.dataframe`, which is a kind of `pandas.DataFrame`, so everything you do with pandas works. To get plain pandas DataFrames instead, set `ssd.options.pandas = True` (or the `STARSIM_DATA_PANDAS` environment variable). The column names are the UN's own: `Time`, `Sex`, `AgeGrpStart` and `mx` are the names used in World Population Prospects, and also the names Starsim's demographic modules expect. Contact matrices are square, with rows and columns labeled by the start age of each age group: ```{python} matrix = ssd.load('kenya', which='prem2021') matrix.iloc[:5, :5] ``` So you can pick out values by age: ```{python} matrix.loc[25, 30] # daily contacts between the 25-29 and 30-34 bins ``` ## Specifying locations You don't need to know the exact name the package uses. ISO codes, official names, common abbreviations, and French, Spanish or Portuguese names are all recognized, and accents, case and punctuation are ignored. ```{python} [ssd.resolve(name) for name in ['USA', 'US', 'united states', 'United States of America', 'Etats-Unis']] ``` ```{python} [ssd.resolve(name) for name in ["Côte d'Ivoire", 'cote divoire', 'Ivory Coast', 'CIV']] ``` A name that doesn't match gives an error rather than a guess, though the error suggests what you might have meant: `ssd.load('Keyna', which='deaths')` gives "location 'Keyna' not found (did you mean: Kenya, China, Ghana?)". [Locations](locations.qmd) has more on which names are recognized. ## Inspecting the data visually Plotting the data before you use it is an easy way to catch the wrong country or a units mistake. ```{python} #| fig-width: 11 #| fig-height: 8 #| out-width: 100% ssd.plot('kenya'); ``` The summary figure shows the age pyramid, the crude birth rate over time, mortality by age and sex, and the contact matrix. Use `which=` to draw one panel on its own. ```{python} #| fig-width: 7 #| fig-height: 5 #| out-width: 100% ssd.plot('kenya', which='deaths', year=2050); ``` ## Attribution Each dataset records its source, license and citation, so you know what to cite: ```{python} prov = ssd.provenance('prem2021') {k: prov[k] for k in ['source', 'license', 'citation']} ``` The same information is attached to every frame returned by `load()`: ```{python} deaths.attrs['provenance'].license ``` [Provenance and reproducibility](provenance.qmd) has more on this. ## Subnational geography Subnational geography is bundled too, as a table of administrative units (provinces, states, districts and so on) with their locations, areas and populations: ```{python} nodes = ssd.load_nodes('kenya', level=1) nodes.head() ``` ```{python} ssd.distances('kenya', level=1).iloc[:4, :4].round(1) ``` Center points, areas and the distances between units need no extra packages and no downloads. Boundary shapes, for drawing maps, are an optional extra described in [Spatial data](spatial.qmd). ## Data that is not bundled The bundled demographic data gives one year per decade, from the UN's Medium projection variant. For every year, complete life tables, or other projection variants, use `download()`, which gets the data from the UN the first time and saves it on your computer: ```python ssd.download('kenya', which='deaths', years=(2000, 2025)) # annual rather than decadal ssd.download('kenya', which='life_table', years=2020) # qx, lx, ex, and the rest ssd.download('kenya', which='births', variant='High') # a different projection variant ``` [Bundled, fetched and downloaded data](access.qmd) explains the difference, and is a good page to read next. ### Bundled, fetched and downloaded data Every dataset in `starsim_data` is one of three kinds, and each kind has its own function, so you can tell from the call where the data comes from: | | Comes from | How you get it | Internet needed | Same for everyone on a version? | |---|---|---|---|---| | **Bundled** | Inside the package | `load()`, `load_nodes()`, `distances()`, `adjacency()` | Never | Yes | | **Fetched** | This package's releases | `fetch()` | First use only | Yes | | **Downloaded** | The original publisher | `download()`, `download_shapes()`, `download_nodes()`, `download_raster()` | First use only | Unless the publisher changes the file | Fetched and downloaded data are both saved on your computer after the first use, so they need the internet only once. The difference is where they come from: `fetch()` gets files from this package's own releases, while `download()` goes to the UN, WorldPop and the boundary publishers. Their servers are outside this package's control, and may be slow, change their files or be down. Calling `fetch()` for data that isn't packaged still works, but warns that it is downloading from an outside source. The `access` column of `list_datasets()` says which kind each dataset is: ```{python} import starsim_data as ssd ssd.list_datasets(detailed=True) ``` ## Bundled The demographic data (one year per decade, Medium variant), the default contact matrices (`prem2021`, plus `prem2017` for the ten locations it lacks), the location table and the subnational node table are inside the package, about 9 MB in all. Everyone who installs a given version has exactly the same data, it works offline, and citing the version is enough to say which data an analysis used. `load()` only ever reads bundled data. If you ask it for anything else, such as `ssd.load('kenya', which='life_table')`, it doesn't download it; it gives an error telling you the `fetch()` or `download()` call to use instead. ## Fetched Some data this package prepares is too large to include, or needed by too few people: - The full `prem2017` and `mistry2021` contact matrices, about 3 MB each. - The US states' contact matrices (`mistry2021_us`, 5 MB) and their populations by single year of age (`population_us`). These are prepared the same way as the bundled data and published with each release of the package. `fetch()` downloads one the first time you use it, checks that it is the file that was published, and saves it. As with bundled data, everyone on a given version gets the same data, and you can list which locations a fetched dataset covers without downloading it. ## Downloaded Everything else comes straight from the organization that publishes it: - Annual UN WPP series, complete life tables, other projection variants and summary indicators, from the UN's bulk files (17–290 MB each). - Administrative boundaries with geometry, from UNOCHA, geoBoundaries or GADM (tens of megabytes per country at admin level 2). - Gridded population rasters from WorldPop (1–40 MB per country-year), and the subnational populations `download_nodes()` computes from them. These are also saved after the first use, along with a note of where and when each was downloaded. Since the files belong to their publishers, a publisher that updates one changes what a fresh download gives you; [Provenance and reproducibility](provenance.qmd) covers how to keep an analysis reproducible anyway. ## The cache Fetched and downloaded files are saved in one folder, the cache: ```{python} ssd.cache_dir() ``` ```python ssd.cache_info() # what is cached, its source, and when it was retrieved ssd.clear_cache() # or clear_cache('wpp/*') ``` To put it somewhere else, set `ssd.options.cache_dir` or the `STARSIM_DATA_CACHE_DIR` environment variable. ## Which kind something is The split is by size and how widely the data is needed, not by subject. The same UN WPP 2024 release is both bundled and downloaded: five series at one year per decade take 4 MB and are bundled, while the complete annual life tables take 290 MB and are downloaded. Likewise, the contact matrices most models should use are bundled (see [Datasets](datasets.qmd#which-source)), and the others are fetched. ```{python} #| echo: false import pandas as pd pd.DataFrame([ dict(requirement="Age structure, mortality, births, fertility, contact matrices", call="load()", access="bundled", cost="instant"), dict(requirement="Positions, areas and separations of administrative units", call="load_nodes(), distances()", access="bundled", cost="instant"), dict(requirement="The other contact matrices (prem2017, mistry2021)", call="fetch()", access="fetched", cost="3 MB each, once"), dict(requirement="US states: contact matrices and populations", call="fetch()", access="fetched", cost="5 MB, once"), dict(requirement="Annual rates, life tables, other variants", call="download()", access="downloaded", cost="17-290 MB, once"), dict(requirement="Polygon geometry, for mapping or spatial joins", call="download_shapes()", access="downloaded", cost="1-50 MB per country"), dict(requirement="Subnational population estimates", call="download_nodes()", access="downloaded", cost="1-40 MB per country"), ]) ``` An analysis that only uses the first two rows needs no internet connection at all, and the package version is enough to say exactly which data it used. ## The data ### Datasets and their shapes This page lists the columns and units of each dataset. The examples are run when this page is built, so they show what the current version actually returns. ```{python} import starsim_data as ssd ssd.list_datasets(detailed='full').drop(columns='license') ``` `layout` is how the data is shaped (the sections below describe each), `levels` the administrative levels it covers (0 for countries, 1 for states or provinces), and `revisions` the WPP revisions it is available for. Licenses are left out here to save space; they are listed in [Provenance](provenance.qmd), and in full in `NOTICE.md`. ## Demographics Five series come from UN World Population Prospects (WPP) 2024, Medium variant, with one year per decade from 1950 to 2100 (sixteen years in all). All 187 countries have all five series. The column names are the UN's own, so the data can be passed to Starsim's demographic modules without renaming anything. ### Births and migration Both are national crude rates, one value per year, not broken down by age. The UN publishes no age breakdown for migration; births by age of mother are the `fertility` series, below. ```{python} births = ssd.load('kenya', which='births') births.head() ``` ```{python} #| echo: false import pandas as pd pd.DataFrame([ dict(column='Time', dtype='int64', units='calendar year', note='decadal, 1950-2100'), dict(column='CBR', dtype='float32', units='births per 1,000 population per year', note=''), ]) ``` The `migration` series has the same shape, with `CNMR` (crude net-migration rate per 1,000) in place of `CBR`. It is signed, with negative values indicating net emigration. ```{python} ssd.load('kenya', which='migration').head(3) ``` ### Deaths Age- and sex-specific mortality, in the usual abridged age groups (`0`, `1–4`, `5–9`, …, `100+`). Only Female and Male are included, not the UN's "Both sexes" rows, to match Starsim. ```{python} deaths = ssd.load('kenya', which='deaths') deaths.head() ``` ```{python} #| echo: false pd.DataFrame([ dict(column='Time', dtype='int64', units='calendar year', note=''), dict(column='Sex', dtype='string', units='Female | Male', note='no "Both sexes"'), dict(column='AgeGrpStart', dtype='int64', units='years', note='start of the bin: 0, 1, 5, 10, ..., 100'), dict(column='mx', dtype='float32', units='deaths per person-year', note='the central death rate m(x)'), ]) ``` ::: {.callout-note} ## Death rates are not probabilities `mx` is the central death rate, deaths per person-year lived, not the probability of dying in the age group (`qx`). The two are similar where mortality is low and differ where it is high. The other life-table columns, including `qx`, are available through `download()`, as described below. ::: ```{python} len(deaths), deaths.Time.nunique(), deaths.AgeGrpStart.nunique() ``` That is sixteen years times two sexes times twenty-two age groups. ### Population Population by 5-year age group and sex, in persons. (The UN publishes these in thousands; they are converted here.) ```{python} population = ssd.load('kenya', which='population') population.head() ``` ```{python} #| echo: false pd.DataFrame([ dict(column='Time', dtype='int64', units='calendar year', note=''), dict(column='Sex', dtype='string', units='Female | Male', note=''), dict(column='AgeGrp', dtype='int64', units='years', note='start of the bin: 0, 5, ..., 100'), dict(column='Value', dtype='float32', units='persons', note='absolute count, not thousands'), ]) ``` The national total is a quick check on the units: ```{python} f"{population[population.Time == 2020].Value.sum() / 1e6:.1f}M" ``` ### Fertility Age-specific fertility rates, by 5-year age group of the mother (`10–14`, …, `50–54`). There is no `Sex` column, since the rates are per woman. ```{python} fertility = ssd.load('kenya', which='fertility') fertility.head() ``` ```{python} #| echo: false pd.DataFrame([ dict(column='Time', dtype='int64', units='calendar year', note=''), dict(column='AgeGrpStart', dtype='int64', units='years', note='start of the bin: 10, 15, ..., 50'), dict(column='ASFR', dtype='float32', units='births per 1,000 women per year', note='the age-specific fertility rate'), ]) ``` Summed over the age groups, times five years per group and divided by 1,000, it gives the total fertility rate: ```{python} f"{5 * fertility[fertility.Time == 2020].ASFR.sum() / 1000:.2f}" ``` ### Revisions The UN updates WPP every two years or so. Each release (revision) is kept separately, so a new one is added alongside the old rather than changing data already in use: ```{python} ssd.list_revisions() ``` ```python ssd.load('kenya', which='deaths') # newest bundled revision ssd.load('kenya', which='deaths', revision=2024) # pinned ``` Naming the revision is a good idea in any analysis you want to be able to reproduce later. ## Contact matrices Contact matrices give the average number of daily contacts between people in each pair of age groups. Three sources are available, each with five contact settings, selected with the `kind` argument. One of them, `prem2021`, is bundled; the other two are fetched the first time you use them (see [Bundled, fetched and downloaded data](access.qmd)). The default, `prem`, is `prem2021`, or `prem2017` for the ten locations `prem2021` lacks (see [Which source](#which-source)). ```{python} matrix = ssd.load('kenya') # prem2021 matrix.iloc[:6, :6].round(3) ``` The matrices are square, with rows and columns labeled by the start age of each age group, so you can pick out values by age: ```{python} matrix.loc[25, 30] ``` ```{python} #| echo: false pd.DataFrame([ dict(source='prem', access='bundled', shape='16x16', bins='5-year, 0-4 ... 75+', countries=187, study='prem2021, with prem2017 for the ten locations it lacks'), dict(source='prem2021', access='bundled', shape='16x16', bins='5-year, 0-4 ... 75+', countries=177, study='Prem et al. 2021'), dict(source='prem2017', access='fetched', shape='16x16', bins='5-year, 0-4 ... 75+', countries=152, study='Prem et al. 2017'), dict(source='mistry2021', access='fetched', shape='85x85', bins='single-year, 0 ... 84+', countries=36, study='Mistry et al. 2021'), dict(source='mistry2021_us', access='fetched', shape='85x85', bins='single-year, 0 ... 84+', countries='51 US states (with DC)', study='Mistry et al. 2021'), ]) ``` The `kind` argument selects the setting: ```{python} {kind: round(float(ssd.load('kenya', kind=kind).to_numpy().sum()), 1) for kind in ('all', 'home', 'work', 'school', 'community')} ``` The `all` matrix is the combined one and the other four are its parts, although they don't add up exactly to `all`, since the original studies estimate them separately. Coverage differs between sources, so it is worth checking. You can list the locations of a fetched source without downloading it: ```{python} len(ssd.list_locations('prem2021')), len(ssd.list_locations('prem2017')), len(ssd.list_locations('mistry2021')) ``` The other two sources are read with `fetch()`, which downloads each one once (about 3 MB) and returns the same kind of frame as `load()`: ```python ssd.fetch('japan', which='prem2017', kind='home') # 16x16 ssd.fetch('france', which='mistry2021') # 85x85, single years of age ``` ### Which source `prem2021` is the authors' own update of `prem2017`, based on more contact surveys and newer demographic data, and covering 177 locations rather than 152. Where both have a country, they agree closely: the median correlation between a country's two matrices is 0.957. `prem2021` dropped ten locations that `prem2017` has: Andorra, Antigua and Barbuda, Australia, Haiti, Japan, Kiribati, Lebanon, Monaco, Seychelles and Taiwan. Their `prem2017` matrices are bundled too, and the default contact matrices, `prem`, use them, so `ssd.load()` has a matrix for all 187 locations. Each matrix records which study it came from: ```{python} ssd.load('Japan').attrs['dataset'] ``` Asking for `prem2021` by name gives that study alone, and the error says where else to find a location it lacks: ```{python} #| error: true ssd.load('Japan', which='prem2021') ``` The full `prem2017` is mainly useful for reproducing earlier analyses. `mistry2021` meets a different need: single-year age groups, but for only 36 mostly high-income countries, under the more permissive Apache-2.0 license. ### US states `mistry2021_us` has the Mistry et al. 2021 matrices for each US state and the District of Columbia, and `population_us` has each one's population in 2023 by single year of age, from the US Census Bureau. A state is named with its country, separated by a colon, as in the node table's `dot_name` (see [Locations](locations.qmd#subnational-units)): ```{python} ssd.list_locations('mistry2021_us')[:3] ``` ```python ssd.fetch('United States:Massachusetts', which='mistry2021_us', kind='school') # 85x85 ssd.fetch('United States:Massachusetts', which='population_us') # Time, Sex, AgeGrp, Value ``` `population_us` has the same columns as `population`, for 2023 only and with `Sex` equal to `'Both sexes'`, since the source does not split by sex. Asking a national dataset for a state tells you where to find it: ```{python} #| error: true ssd.load('United States:Massachusetts') ``` ## The node table The node table has one row per administrative unit (province, state, district and so on). It is described in full in [Spatial data](spatial.qmd); its columns are listed here too. ```{python} nodes = ssd.load_nodes('kenya', level=1) nodes.head(3) ``` ```{python} #| echo: false pd.DataFrame([ dict(column='nodeid', dtype='int32', note='numbered from 0 within each country and level; matches the rows of distances()'), dict(column='location', dtype='str', note='the country, e.g. Kenya'), dict(column='iso3', dtype='str', note='ISO 3166-1 alpha-3'), dict(column='level', dtype='int32', note='0 = country, 1 = first-level subdivision'), dict(column='name', dtype='str', note="the unit's own name, as the source spells it"), dict(column='pcode', dtype='str', note='a standard code for the unit, for joining to other data: a p-code from UNOCHA, or a geoBoundaries ID'), dict(column='dot_name', dtype='str', note='hierarchical path, e.g. Kenya:Baringo'), dict(column='parent', dtype='str', note="the parent's dot_name; empty at level 0"), dict(column='lat', dtype='float64', note='latitude of the center of the unit'), dict(column='lon', dtype='float64', note='longitude of the center of the unit'), dict(column='area_km2', dtype='float64', note='area in square kilometers'), dict(column='population', dtype='float64', note='UN WPP at level 0; WorldPop 2020 summed within each unit at level 1'), dict(column='source', dtype='str', note='where the boundaries came from: unocha or geoboundaries'), ]) ``` ## Beyond the bundle For the contact sources that are not bundled (`prem2017`, `mistry2021` and `mistry2021_us`), `fetch()` returns the same kind of frame `load()` returns for `prem2021`. For the five bundled demographic series, `download()` returns the same columns for every year, straight from the UN, and it also provides two datasets that are not bundled at all: ```{python} ssd.list_fetchable(), ssd.list_downloadable() ``` ```python ssd.download('kenya', which='deaths', years=(2000, 2025)) ``` | Returns | Columns | |---|---| | `births`, `migration` | as bundled, but every year | | `deaths`, `population`, `fertility` | as bundled, but every year | | `life_table` | `Time, Sex, AgeGrpStart, mx, qx, px, lx, dx, Lx, Sx, Tx, ex, ax`, for Female, Male and `Total` | | `indicators` | `Time` plus ~50 WPP indicators: `CDR`, `TFR`, `LEx`, `IMR`, `Q5`, `NetMigrations`, … | The `life_table` result comes with a glossary of its columns, since the standard two-letter names are easy to mix up: ```python lt = ssd.download('kenya', which='life_table', years=2020) lt.attrs['columns']['Lx'] # 'person-years lived between ages x and x+n' ``` ## What is not here - **Subnational demographics,** other than the populations of the US states (`population_us`, above). WPP has no subnational dimension, so none can be derived from it. - **Age-specific migration.** WPP publishes only the crude national rate. - **"Both sexes" rows** in the bundled data, which has Female and Male only. `download(..., which='life_table')` does include `Total`. - **Hong Kong, Macao and Puerto Rico** in the node table, since no boundary source treats them separately. Their demographics and contact matrices are included. ### Locations Each location has one standard name in this package, such as `Kenya`, `United_States` or `Cote_dIvoire`, but people write country names in many other ways. This page covers which names are recognized and what happens when one isn't. ```{python} import starsim_data as ssd [ssd.resolve(name) for name in ['Kenya', 'kenya', 'KE', 'KEN', 'Republic of Kenya']] ``` ## What is recognized These forms are all recognized: | Form | Example | Resolves to | |---|---|---| | Standard name | `United_States` | `United_States` | | Free-text name | `united states` | `United_States` | | ISO 3166-1 alpha-2 | `US` | `United_States` | | ISO 3166-1 alpha-3 | `USA` | `United_States` | | Official name | `United States of America` | `United_States` | | Common abbreviation | `DRC`, `UK`, `UAE` | the corresponding country | | Former name | `Zaire`, `Burma` | the current country | | French, Spanish or Portuguese name | `Allemagne`, `Alemania` | `Germany` | Accents, case, punctuation and spacing are ignored, so these are all treated as the same name: ```{python} {name: ssd.locations.normalize(name) for name in ["Côte d'Ivoire", "cote divoire", "Cote_dIvoire", " COTE D'IVOIRE "]} ``` ```{python} len(set(ssd.resolve(n) for n in ["Côte d'Ivoire", 'cote divoire', 'Ivory Coast', 'CIV'])) ``` ## Resolution is exact; approximate matching only advises A name has to match one in the table exactly (after ignoring accents, case and punctuation). Close matches are only used to suggest names in the error message; the package never guesses which country you meant. ```{python} #| error: true ssd.load('Keyna', which='deaths') ``` Guessing would go wrong both ways. Abbreviations like `USA` look nothing like `United_States`, so they have to be listed in the table rather than guessed. And a name that is no country at all still has close matches: ```{python} import sciris as sc sc.suggest('Atlantis', ssd.list_all_locations(), n=3, die=False) ``` That is helpful in an error message, but would be wrong as an answer. ## Subnational units A subnational unit is named by its country and its own name, separated by a colon, as in the node table's `dot_name`. The country part is recognized as above; the unit's name has to match exactly, ignoring case, accents and punctuation: ```{python} [ssd.resolve(name) for name in ['United States:Massachusetts', 'USA:massachusetts', 'us:new york']] ``` When no dataset is given, the units are those of the node table, and a misspelled unit gets suggestions from its own country. Only a few datasets have subnational units so far: `mistry2021_us` and `population_us`, for the US states (see [Datasets](datasets.qmd#us-states)). ## Scoping to a dataset Coverage varies between datasets, so `load()` checks the name against the locations in the dataset you asked for. A country that exists but is missing from that dataset gives a different error from a name that isn't a country at all: ```{python} #| error: true ssd.load('Japan', which='prem2021') ``` ```{python} len(ssd.list_locations('prem2021')), len(ssd.list_locations('mistry2021')) ``` `list_locations(which)` gives the locations in one dataset, and `list_all_locations()` gives every location the package knows about. ```{python} len(ssd.list_all_locations()) ``` ## Inspecting the table ```{python} info = ssd.location_info('DRC') {k: info[k] for k in ['location', 'iso_code', 'iso3', 'official_name']} ``` ```{python} ssd.location_info('Germany').aliases[:8] ``` The table itself is `starsim_data/data/locations.csv`, with official names and aliases taken from the `pycountry` package. Any alias that could mean two different locations is left out, so a name that matches always means just one. ::: {.callout-warning} ## Namibia's country code Namibia's two-letter ISO code is `NA`, which `pandas.read_csv` reads as a missing value by default. This package handles it, but if you read country tables with pandas yourself, pass `keep_default_na=False` to avoid losing Namibia. ```{python} ssd.resolve('NA'), ssd.location_info('Namibia').iso3 ``` ::: ## Administrative levels `parse_level()` accepts the common ways of writing an administrative level and returns a number: ```{python} [ssd.parse_level(x) for x in ['admin1', 'ADM2', '3', 4, 'level0']] ``` Level 0 is the country, level 1 its first subdivisions (provinces, states), and so on. The highest is 5, the finest level any of the boundary sources in [Data sources](datasources.qmd) publishes. ```{python} #| error: true ssd.parse_level('admin9') ``` `True` and `False` are rejected, rather than quietly read as levels 1 and 0: ```{python} #| error: true ssd.parse_level(True) ``` ## Naming below the country level Subnational units in the node table are named by their `dot_name`, the country and unit names joined by a colon, cleaned up the same way but keeping capitals: ```{python} ssd.load_nodes('France', level=1).dot_name.head(4).tolist() ``` ```{python} [ssd.node_name(x) for x in ['New York', 'Île-de-France', 'KwaZulu-Natal']] ``` The part before the colon is always a name `load()` accepts, so you can split a `dot_name` and look up its country directly. To join to other data, though, use `pcode` rather than `dot_name`; see [Spatial data](spatial.qmd). ### Spatial data Spatial data comes at two levels of detail. The **node table** describes each administrative unit (a province, county or district) by its location, area, population and an identifier, and records which units share a border. That is enough for most metapopulation models, and it is small enough to be bundled with the package. **Boundary shapes** and **population grids** are much larger, so they are downloaded from their publishers when you ask for them, and need some optional extra libraries. ## The node table ```{python} import starsim_data as ssd nodes = ssd.load_nodes('kenya', level=1) nodes.head() ``` Level 0 returns the country itself, and omitting `level` returns every level at once: ```{python} ssd.load_nodes('kenya')[['name', 'pcode', 'population', 'area_km2']] ``` The table covers levels 0 and 1 for 184 countries: ```{python} len(ssd.list_locations('nodes')) ``` Hong Kong, Macao and Puerto Rico are missing, since none of the boundary sources treats them as separate countries. Their demographics and contact matrices are still available. ### Identifiers Two columns identify a unit, and they serve different purposes. `pcode` is the code to join other data on. For units from UNOCHA it is the unit's p-code: a standard code (such as `KE047` for Nairobi) that the UN and most humanitarian and health datasets use to label subnational areas. That means you can join the node table to case counts, campaign coverage or survey data by code rather than by matching names. For units from geoBoundaries it is a geoBoundaries ID instead, which works the same way within this package but won't match p-coded data from elsewhere. `dot_name` is a readable name that includes the parent units, like `Kenya:Nairobi`. Use it for display, for referring to units in configuration files, and for grouping; the part before the first colon is always a location that `load()` accepts. ```{python} nodes[['name', 'pcode', 'dot_name', 'parent', 'source']].head(3) ``` ### Geometry-derived columns `lat` and `lon` are the center point (centroid) of the unit, in ordinary latitude and longitude. `area_km2` is the unit's area, measured on the curved surface of the Earth rather than on a flat map. Adding up a country's subdivisions gives the country's total to within a small fraction of a percent: ```{python} country = ssd.load_nodes('Kenya').area_km2.iloc[0] counties = ssd.load_nodes('Kenya', level=1).area_km2.sum() round(country), round(counties), round(100 * abs(country - counties) / country, 3) ``` ### Population `population` at level 0 is the country's UN WPP population for a single reference year: ```{python} ssd.provenance('nodes').population_year ``` At level 1 it is WorldPop's 2020 population grid (1 km squares, adjusted to match UN totals), added up within each unit. This is worked out when the package is built (see [Building the data](building-the-data.qmd#the-node-table)), so you need no download and no extra libraries to use it: ```{python} ssd.load_nodes('kenya', level=1).sort_values('population', ascending=False)[['name', 'population']].head(3) ``` Other years and finer levels are a download; see [subnational population](#subnational-population) below. ### Mixed sources Boundaries come from UNOCHA where it covers the country, and from geoBoundaries otherwise. The two don't always agree on how a country is divided up, so each row records which source it came from: ```{python} set(ssd.load_nodes('Kenya', level=1).source), set(ssd.load_nodes('France', level=1).source) ``` Using different sources for different countries in one analysis is usually fine. Switching sources for the same country between analyses is not, since a change of source can change the number of units, so check the `source` column before comparing. ## Distances ```{python} d = ssd.distances('kenya', level=1) d.iloc[:4, :4].round(1) ``` The matrix is square and symmetric, with zeros on the diagonal, and is labeled by `dot_name` by default. Distances are straight-line distances over the Earth's surface between the units' centroids. They are accurate to about half a percent, much less than the error from treating a whole province as a single point. ```{python} round(float(d.loc['Kenya:Nairobi', 'Kenya:Mombasa']), 1) # km ``` You can choose miles or kilometers and which column to label by, and you can pass in your own node table (for example, one with updated populations or only some of the units): ```{python} ssd.distances('Kenya', level=1, units='mi', index='pcode').iloc[:3, :3].round(1) ``` Together with the population column, this gives you the inputs for a gravity model. For the standard form $$c_{ij} = k \, \frac{P_i^{a} P_j^{b}}{D_{ij}^{c}}$$ the node table supplies $P_i$ and $P_j$ and `distances()` supplies $D_{ij}$. The gravity model itself belongs in your model, so this package doesn't implement it. ## Borders `adjacency()` says which units share a border, as a square true/false matrix labeled like `distances()`. Units within about 100 m of each other count as touching, which allows for small gaps between neighboring boundaries. It is bundled, so it needs no download or extra libraries: ```{python} a = ssd.adjacency('kenya', level=1) sorted(a.columns[a.loc['Kenya:Nairobi']]) ``` Use it for a model that moves people only between neighbors, or to check a distance-based model against the map. ## Optional dependencies Everything above requires only the base installation. The remainder of this page requires the `[spatial]` extra: ```bash pip install "starsim_data[spatial]" ``` This installs `geopandas`, `pyogrio` and `rastertoolkit`. Without them, the rest of the package works as normal, and only the functions that need them give an error: ```python ssd.download_shapes('kenya') # ImportError: download_shapes() needs the optional geospatial dependencies; # install them with: pip install starsim_data[spatial] ``` ## Boundaries with geometry ```python gdf = ssd.download_shapes('kenya', level=1) ``` The result is a `GeoDataFrame` (a dataframe with a column of boundary shapes) with the columns `nodeid`, `location`, `iso3`, `level`, `name`, `pcode`, `dot_name`, `source` and `geometry`. The `pcode` and `dot_name` values match the node table for the same country and level, so you can join the two on either. Three sources are available: ```{python} ssd.list_shape_sources() ``` With no `source` argument, UNOCHA is tried first and geoBoundaries second. UNOCHA comes first because of its p-codes; geoBoundaries covers the countries UNOCHA doesn't, and this package uses a fixed geoBoundaries release so the boundaries don't change under you. ::: {.callout-warning} ## GADM is licensed for non-commercial use only GADM is never used automatically: you have to ask for it by name, with `source='gadm'`. Nothing derived from it is bundled with this package or redistributed by it. Data you download from GADM is subject to its terms, and following them is up to you. ::: ## Subnational population `download_nodes()` returns the node table with `population` filled in for any year from 2000 to 2030, by downloading WorldPop's 1 km population grid for the country and adding it up within each unit: ```python nodes = ssd.download_nodes('kenya', level=1, year=2025) ``` At levels 0 and 1, it uses the same boundary source as the bundled node table, so the units match. At finer levels, it builds the table from the downloaded boundaries, including which units share a border: ```python subcounties = ssd.download_nodes('kenya', level=2) # Kenya's 290 sub-counties ssd.adjacency('kenya', level=2, nodes=subcounties) # their borders, from the same polygons ``` The two steps are also available separately, if you want to work with the grid or the boundaries yourself: ```python raster = ssd.download_raster('kenya', year=2020) shapes = ssd.download_shapes('kenya', level=1) totals = ssd.aggregate_raster(raster, shapes, by='pcode') ``` ::: {.callout-note} ## WorldPop and UN WPP totals differ WorldPop's figures are modeled estimates rather than census counts, and they don't add up exactly to the UN WPP national total for the same year. For Kenya in 2020, the WorldPop total is about 1.2% below the WPP figure. Neither is wrong; they estimate slightly different things in different ways. If your model needs the subnational figures to add up to the national one, `rescale_to_national()` adjusts them: ```python nodes = ssd.rescale_to_national(ssd.download_nodes('kenya', level=1)) ``` ::: WorldPop's server is slow, so the first request for a country can take a while; after that the file is saved on your computer and reused. For some very small territories WorldPop has no data, and you'll get an error saying so. ## Mapping There are two options here as well. `plot(which='nodes')` draws the centroids from the bundled table and needs no extra libraries: ```{python} #| fig-width: 7 #| fig-height: 6 #| out-width: 100% ssd.plot('kenya', which='nodes', level=1); ``` `choropleth()` draws filled-in boundary maps, and needs the `[spatial]` extra: ```python ssd.choropleth('kenya', level=1) # colored by downloaded population ssd.choropleth('kenya', level=1, shapes=my_frame, value='incidence') ``` Passing your own `shapes` lets you map any column, which is the usual way to map model results. ### Data sources The sources this package uses cover different sets of countries, are updated on different schedules, come under different licenses, and sometimes disagree with one another. This page compares them: what each is good for, its limitations, and how to choose between them. Much of the material on boundaries and population grids is adapted from [`laser-init`](https://github.com/laser-base/laser-init); see [Related projects](related-projects.qmd). The full attribution and license text is in [NOTICE.md](https://github.com/starsimhub/starsim_data/blob/main/NOTICE.md). You can also get each dataset's source, license and citation in Python with `ssd.provenance(which)`, or from `df.attrs['provenance']` on any dataframe the package returns; see [Provenance and reproducibility](provenance.qmd). ## What comes from where | Data | Source | Access | How you get it | |---|---|---|---| | Births, deaths, migration, population | UN WPP 2024 | bundled | `load()` | | Contact matrices | Prem 2017 / Prem 2021 / Mistry 2021 | bundled (`prem`), fetched (the rest) | `load()` | | Node table: centroids, areas, populations, p-codes | UNOCHA COD-AB, geoBoundaries | bundled | `load_nodes()` | | Annual series, life tables, other variants | UN WPP 2024 bulk files | downloaded | `download()` | | Administrative boundaries with geometry | UNOCHA, geoBoundaries, GADM | downloaded | `download_shapes()` | | Gridded population | WorldPop | downloaded | `download_raster()`, `download_nodes()` | Bundled data is inside the package, works offline, and is the same for everyone using the same version. Fetched data is published with this package's releases and downloaded the first time you use it. Downloaded data comes straight from its original publisher. Fetched and downloaded files are saved on your computer, so each is downloaded only once. What goes where depends mostly on size, which is why some UN WPP data is bundled and the rest is downloaded; see [Bundled, fetched and downloaded data](access.qmd). --- ## Demographics ### UN World Population Prospects **Provider:** UN Department of Economic and Social Affairs, Population Division · [population.un.org/wpp](https://population.un.org/wpp/) · CC BY 3.0 IGO The standard national demographic series, covering all countries from 1950 to 2100 and revised every two years. | | | |---|---| | **Coverage** | 187 countries (regional totals are left out) | | **Levels** | National only | | **Variants** | Medium (bundled), plus High, Low, Constant fertility and others via `download()` | | **Update** | Every two years: 2022, 2024, … | **Strengths.** The estimates use the same methods for every country, so countries can be compared, and the time series is long and consistent. The methods are well documented, and the data is revised often enough to stay current. **Limitations.** The data is national only, so you can't get province-level rates from it. Projections are scenarios rather than predictions. Estimates can change a lot between revisions, since a new census can revise a country's past figures too, which is why you can pick a revision with `load(..., revision=2024)`. **Two ways to get it.** The bundled data, which `load()` reads, covers every tenth year and is small enough to ship with the package. `download()` reads the UN's own files instead, for annual data, complete life tables and other projection variants; each file is a 17–290 MB download, needed once. **If you read WPP files yourself:** Namibia's two-letter code is `NA`, which pandas reads as a missing value unless you pass `keep_default_na=False`. This package handles it for you. --- ## Contact matrices Three sources are available. `prem2021` is bundled; the other two are fetched the first time you use them. They differ mainly in age groups and country coverage. The default contact matrices, `prem`, are `prem2021`, filled in with `prem2017` for the ten locations `prem2021` lacks; see [Datasets](datasets.qmd#which-source). | Source | Access | Matrix | Age bins | Countries | License | |---|---|---|---|---|---| | `prem2021` | bundled | 16×16 | 5-year, 0–75+ | 177 | GPL-3.0 | | `prem2017` | fetched | 16×16 | 5-year, 0–75+ | 152 | GPL-3.0 | | `mistry2021` | fetched | 85×85 | single-year, 0–84+ | 36 | Apache-2.0 | | `mistry2021_us` | fetched | 85×85 | single-year, 0–84+ | 51 US states (with DC) | Apache-2.0 | **Prem et al. 2017 and 2021** estimate contact patterns for countries without their own contact survey, based on household composition and age structure. The 2021 paper is an update that covers more countries and compares the results with survey data; use it unless you are reproducing earlier work. Both provide the same five settings: `all`, `home`, `work`, `school` and `community`. **Mistry et al. 2021** has single-year age groups up to 84, but for only 36 countries. Use it when five-year age groups are too coarse, for example for school-entry effects, and your country is covered. The same study's matrices for the US states are `mistry2021_us`; each reflects that state's own households, schools and workplaces rather than the national average. Not every country is in every source, so check with `list_locations('prem2021')` (or the source you want). The matrices were adapted from [epydemix-data](https://github.com/epistorm/epydemix-data), which also has subnational matrices for other countries and US counties that aren't included here; see [Related projects](related-projects.qmd). ### US Census Bureau **Provider:** US Census Bureau, Population Estimates Program · [census.gov/popest](https://www.census.gov/programs-surveys/popest.html) · public domain `population_us` is the Vintage 2023 estimate of each state's population by single year of age, as compiled by epydemix-data. It covers the 50 states and the District of Columbia (not the territories), for 2023 only, both sexes together, with the last age group 84+ to match `mistry2021_us`. Population totals for other years, or by sex, are not included. --- ## Administrative boundaries The three boundary sources complement each other, which is why all three are available and why `download_shapes()` tries more than one. ### UNOCHA COD-AB **Provider:** UN Office for the Coordination of Humanitarian Affairs · downloaded from [laser-base's copy](https://github.com/laser-base/unocha) · CC BY 3.0 IGO | | | |---|---| | **Coverage** | ~110 countries, weighted towards humanitarian operations | | **Levels** | 0–4, varying by country | | **Download size** | 0.2–5 MB per country | | **Key** | **P-code** | **Strengths.** The p-codes are the reason this source is tried first. A p-code is the standard code the UN uses for each subnational area, and most humanitarian and health data uses it too, including surveillance line lists, campaign coverage and datasets on [HDX](https://data.humdata.org), so you can join that data to these boundaries by code. The boundaries are usually the official government ones, checked through UN operations, and are updated quickly during an emergency response. **Limitations.** It covers about 110 countries, mostly those with humanitarian operations; many middle- and high-income countries are missing. ### geoBoundaries **Provider:** William & Mary geoLab · [geoboundaries.org](https://www.geoboundaries.org) · CC BY 4.0 | | | |---|---| | **Coverage** | Near-universal (~200 countries) | | **Levels** | 0–5, varying by country | | **Download size** | 1–50 MB per country | | **Key** | `shapeID` (not a p-code) | **Strengths.** It covers the countries UNOCHA doesn't. It is published in numbered releases, and this package uses a fixed one (v6.0.0), so the boundaries don't change from one run to the next. The methods are peer-reviewed and documented. **Limitations.** Quality varies more by country and level than GADM's does. It uses its own IDs rather than p-codes, so joining p-coded data needs a lookup table between the two. Recent boundary changes may not be included yet. ### GADM **Provider:** Global Administrative Areas · [gadm.org](https://gadm.org) · **non-commercial use only** | | | |---|---| | **Coverage** | Universal | | **Levels** | 0–5, the deepest hierarchies of the three | | **Download size** | 1–100 MB per country (all levels in one file) | | **Key** | `GID_n` | **Strengths.** The most complete and most detailed of the three, with names in several languages and frequent updates. **The license is the catch.** GADM is free for non-commercial use only, which rules it out for much applied public-health work. So this package: - never uses GADM automatically: you have to ask for it with `download_shapes(..., source='gadm')`, which prints a reminder of the terms; - never redistributes anything derived from it: the bundled node table is not built from GADM. If you download GADM data, following its terms is up to you. ### Choosing between them With no `source` argument, `download_shapes()` tries UNOCHA and then geoBoundaries. That suits most disease modeling: p-codes where they exist, and coverage everywhere else. Choose a source yourself when: - **you need level 4–5** — geoBoundaries or GADM; - **you are reproducing a published analysis** — whichever it used; - **you need a country's official boundaries** — check all three against the national statistical office, since they do disagree; - **you are joining to p-coded data** — UNOCHA; if UNOCHA lacks the country, you'll need a lookup table between p-codes and the other source's IDs. One caution: the sources don't always agree on how many units a country has, or where the boundaries are. Using different sources for different countries is usually fine; switching sources for the same country over time usually is not. The node table records the `source` of each row for this reason. --- ## Gridded population ### WorldPop **Provider:** WorldPop, University of Southampton · [worldpop.org](https://www.worldpop.org) · CC BY 4.0 | | | |---|---| | **Coverage** | Global, with gaps for very small territories | | **Resolution** | 1 km (this package); 100 m available upstream | | **Years** | 2000–2030 across two product series | | **Download size** | 1–40 MB per country and year | | **Method** | Census counts spread over a grid using a statistical model (satellite imagery, settlements, roads and so on) | WorldPop estimates how many people live in each 1 km square. Adding those up within each administrative unit gives a population for each node, which `download_nodes()` does for you. **Strengths.** Global, open, detailed, adjusted to UN totals, and widely validated. Because there is a series of years, a model's population distribution can change over time rather than staying fixed. **Limitations.** These are modeled estimates, not census counts. They don't add up exactly to the UN WPP national total for the same year; for Kenya in 2020 the WorldPop total is about 1.2% lower. Neither figure is wrong, as they estimate slightly different things. If your model needs the national total to match WPP, use `rescale_to_national()`. Estimates are also more accurate in cities than in rural areas, and come without uncertainty ranges. WorldPop's server is slow, so the first download for a country can take a while; after that it is saved on your computer and reused. --- ## Reproducibility The original publishers revise their data over time. To keep an analysis reproducible, pin the `starsim_data` version, pick the WPP revision explicitly, and record the boundary source you used. [Provenance and reproducibility](provenance.qmd) has the details. --- ## Not included These aren't included; here is where else to look: - **Subnational demographics.** WPP provides none. National statistical offices, DHS and IPUMS are the usual alternatives. - **Age-specific migration.** WPP publishes the crude national rate only. - **Other population grids** — LandScan, GHSL, GRID3, Meta HRSL. WorldPop was chosen because it is open and globally consistent; the others are better for some questions. - **OpenStreetMap and Natural Earth boundaries.** OpenStreetMap's quality varies a lot from place to place; Natural Earth is simplified for drawing maps rather than for analysis. - **Health-system, mobility and climate data.** Out of scope for this package. ### Related projects Almost all the data in this package is public, and other projects package it for modelers too. What differs between them is mostly how you get it: whether the data comes with the package or from a server, whether you get a dataframe or a folder of files, and how much of the model is set up for you. Two projects in particular shaped this one. For the original data sources themselves (coverage, updates, licenses and limitations), see [Data sources](datasources.qmd); this page is about the packages that sit between those sources and a model. ## The two this package builds on ### epydemix-data [epydemix-data](https://github.com/epistorm/epydemix-data) is the data behind [epydemix](https://github.com/epistorm/epydemix), the ISI Foundation's Python package for stochastic compartmental modeling and calibration. It has age distributions and contact matrices for more than 3,500 regions, as folders of CSV files on GitHub. **What this package took from it.** The contact matrices here (`prem2017`, `prem2021`, `mistry2021` and `mistry2021_us`, plus the US state populations) come from epydemix-data's carefully cleaned-up files, keeping the national and US-state ones. Assembling them from the papers' original spreadsheets is tedious work, and epydemix-data had already done it well. This package also borrowed the idea of a single list of locations and the data available for each, and of tying each package version to an exact version of the data. **Where the two differ.** epydemix-data reads its files from GitHub by default, unless you have downloaded a local copy. `ssd.load()` works offline, because the data comes with the package. Each choice has its advantages: a collection of files on GitHub can hold 3,500 regions and four contact sources, which a package can't, while bundled data is instant, works offline and is the same for everyone using the same version. The scope differs in both directions: - **epydemix-data has more contact data:** state and province matrices for other countries, all 3,144 US counties, a fourth source (`litvinova_2025`), and US matrices by sex and by race/ethnicity. This package has national and US-state matrices only. - **This package has more demographic data.** epydemix-data has one age distribution per location, which is what a compartmental model needs to get started. Here there are five UN WPP series from 1950 to 2100 (births, deaths by age and sex, migration, population, and fertility by age), because an agent-based model with births and deaths needs rates over time, not just a starting age structure. - **This package adds geography.** epydemix-data has no boundaries, node table or population grids. The licenses differ too. epydemix-data is GPL-3.0 as a whole. Here the code is MIT and each dataset keeps its original license (`ssd.provenance(which).license`): the Prem matrices are GPL-3.0, `mistry2021` is Apache-2.0, and the WPP data is CC BY 3.0 IGO. See [NOTICE.md](https://github.com/starsimhub/starsim_data/blob/main/NOTICE.md) for details. ### laser-init [laser-init](https://github.com/laser-base/laser-init) sets up a spatial model for [LASER](https://github.com/laser-base/laser-generic). It is a command-line tool rather than a library: `laser-init NGA 2 2000 2025` downloads boundaries, WorldPop population and UN WPP data for Nigeria at admin level 2, works out the population of each unit, and writes a folder with the boundaries, demographic CSVs, a config file, plots, a PDF report, a record of where the data came from, and a runnable SEIR model script. **What this package took from it.** The spatial data here follows laser-init's approach: the three boundary sources and when to prefer each, trying UNOCHA then geoBoundaries with GADM only on request, and adding up WorldPop's population grid within each unit. Much of the boundary and population material in [Data sources](datasources.qmd) is adapted from laser-init's documentation. Keeping a record of where each piece of data came from was also its idea. **Where the two differ.** laser-init sets up a working model; this package returns dataframes and stops there. laser-init also downloads everything on its first run, which with its default boundary source is 1–2 GB. Here the node table (centroids, areas, populations, p-codes and borders for 184 countries at levels 0 and 1) is bundled, which covers most of what a metapopulation model needs with no download and no extra libraries. Boundary shapes, finer levels and population grids are still downloaded when you need them. laser-init's demographics are birth and death rates and a survival curve for one country and period, whereas `load()` returns the full WPP series for every country. Both are MIT-licensed. ### The three side by side | | `starsim_data` | `epydemix-data` | `laser-init` | |---|---|---|---| | **Form** | Python library | Git-hosted CSV tree + loader in `epydemix` | Command-line tool | | **You get** | Dataframes | A `Population` object | A directory of files and a model script | | **Internet needed** | No, for bundled data | Yes, unless downloaded first | Yes | | **Demographics** | Five UN WPP series, 1950–2100 | One age distribution per location | Birth and death rates, age distribution, survival curve | | **Contact matrices** | 3 sources, national and US state | 4 sources, national to US county | None | | **Geography** | Node table bundled; boundaries and population grids downloaded | None | Boundaries and population grids downloaded | | **Locations** | 187 countries, 51 US states | 3,500+ regions | Any one country per run | | **Generates a model** | No | No | Yes | | **License** | MIT code, per-dataset data licenses | GPL-3.0 | MIT | Each is best for something different. For a compartmental model of a US county, or of another country's provinces, epydemix-data has contact data this package doesn't. For a spatial LASER model of Nigeria running as soon as possible, laser-init will get you there faster than building one from `load_nodes()`. This package is aimed at Starsim models that need national demographic rates and contact matrices, offline and reproducibly, with subnational geography when you want it. ## Other places these data come from Other projects that package data for epidemic modeling: ### Contact matrices | Project | Language | What it provides | |---|---|---| | [`contactdata`](https://github.com/socialcontactdata/contactdata) | R (CRAN) | The Prem 2017 and 2021 matrices for 177 countries. The closest R equivalent to this package's contact matrices, and a good choice if matrices are all you need. | | [`socialmixr`](https://github.com/epiforecasts/socialmixr) | R (CRAN) | Contact matrices computed directly from contact surveys (POLYMOD and the wider [socialcontactdata.org](http://socialcontactdata.org) collection), rather than estimated for countries without a survey. Covers a few dozen surveys rather than every country. | | [`mobs-lab/mixing-patterns`](https://github.com/mobs-lab/mixing-patterns) | Data repository | The original Mistry 2021 matrices and the synthetic populations they came from. The source of both `mistry2021` here and epydemix-data's copy. | | [`synthpops`](https://github.com/synthpops/synthpops) | Python | Builds synthetic populations with household, school and workplace networks, rather than providing matrices. A contact matrix summarizes who mixes with whom; a synthetic population spells it out person by person. | The main difference in this group is **estimated versus measured**. Prem 2017 and 2021 estimate contact patterns for countries without their own survey, from household composition and age structure; `socialmixr` computes matrices only where a survey exists. The estimates cover almost every country and are what most models use; measured matrices are better where a survey exists and the choice of matrix matters for your results. ### Demographics | Project | Language | What it provides | |---|---|---| | [`wpp2024`](https://github.com/PPgp/wpp2024) and [`wpp2024extra`](https://github.com/PPgp/wpp2024extra) | R (GitHub) | The UN WPP 2024 release as R data packages, including annual data by single year of age and projection uncertainty ranges. The most complete packaging of WPP in any language; this package's `download()` covers part of the same ground. | | [UN WPP Data Portal](https://population.un.org/dataportal/) | API | The official source; needs a free access token. `download()` uses the UN's public files instead, so you don't need one. | | National statistical offices, [DHS](https://dhsprogram.com), [IPUMS](https://international.ipums.org) | Various | Subnational demographics, which WPP doesn't have. If your model needs province-level fertility or mortality, look here. | ### Geography and population | Project | Language | What it provides | |---|---|---| | [geoBoundaries](https://github.com/wmgeolab/geoBoundaries) | Data + API | Open administrative boundaries for nearly every country. Used directly by `download_shapes()`. | | [HDX](https://data.humdata.org) and [`hdx-python-api`](https://pypi.org/project/hdx-python-api/) | Python | The Humanitarian Data Exchange, home of the UNOCHA boundaries and most subnational health data labeled with p-codes. The node table has p-codes so you can join it to data from here. | | [`pygadm`](https://pypi.org/project/pygadm/) | Python | GADM boundaries, which go down to the finest administrative levels. Note the non-commercial license, which is why this package downloads GADM only on request and never redistributes it. | | [WorldPop](https://hub.worldpop.org) and [`rasterstats`](https://pypi.org/project/rasterstats/) | Data + Python | Population grids, and a tool for adding them up within boundaries. `download_nodes()` does this for you; `rasterstats` is the general tool if you need something else. | ## Choosing between them Most of these projects use the same underlying data, so a few rules of thumb: - **Use whichever is closest to your modeling framework.** They mostly draw on the same few sources, so the difference is convenience rather than accuracy: pick the one that gives you data in the form your model wants. - **Don't mix packages for the same data in one analysis.** Two packages may both offer "the Prem 2021 matrix for Kenya" but differ in age-group labels, scaling or which release they use. Pick one and note which. - **Check what has been left out.** Every package here includes only part of its sources. This one leaves out subnational contact matrices, for example, and bundles only the Medium WPP projection; [Data sources](datasources.qmd) lists what is missing. - **Record what you used.** Whichever you choose, note the version and where the data came from; see [Provenance and reproducibility](provenance.qmd). ## Internals ### Architecture This page describes how the package is organized internally: what the modules do, how a call to `load()` is dispatched, how data is stored on disk, and where the extension points are. It is intended for people modifying the package or adding data to it, rather than for people using it. ## Modules ```{python} #| echo: false import pandas as pd pd.DataFrame([ dict(module='settings.py', role='The `options` object: cache location and verbosity'), dict(module='registry.py', role='What datasets exist, how each is shaped, and the registration hooks'), dict(module='locations.py', role='Resolving location strings; administrative-level vocabulary'), dict(module='loaders.py', role='Reading bundled stores: `load`, `list_datasets`, `provenance`'), dict(module='cache.py', role='The download cache and its provenance sidecars'), dict(module='fetching.py', role='Stores published with the releases: `fetch`'), dict(module='downloading.py', role='UN WPP bulk files: `download`'), dict(module='nodes.py', role='The node table and `distances`'), dict(module='shapes.py', role='Boundary sources and `download_shapes` (needs [spatial])'), dict(module='rasters.py', role='WorldPop rasters and `download_nodes` (needs [spatial])'), dict(module='plotting.py', role='Validation figures and the choropleth'), ]) ``` Import order matters, because two modules register themselves when they are imported: `loaders` adds its layout shapers and `downloading` adds its tier-2 datasets, both of which require `registry` to exist first. The order is fixed in `__init__.py`. ## Import conventions Modules follow Starsim's convention: each imports the package under its own alias and reaches everything through it. ```python import starsim_data as ssd def load_nodes(location, level=0): return ssd.load(location, which='nodes', level=level) ``` `from` imports are confined to `__init__.py`, where the public namespace is assembled. Within a module, `ssd.load` and `ssd.registry.datasets()` resolve at call time, so modules can refer to each other freely. ## The registry The registry holds two tables. `DATASETS` maps a dataset name to a descriptor recording its layout, tier, administrative levels, license, available revisions, location count and where its store lives. ```{python} import starsim_data as ssd ssd.registry.dataset('deaths') ``` Descriptors for bundled data are discovered once, by opening each `.h5` file in the data directory and reading its root attributes. The result is cached; `registry.refresh()` discards the cache, which is needed only if the files change within a running process, such as after a rebuild in a test. The default contact matrices, `prem`, are described at the same time, from `prem2021.h5` and the ten matrices in `prem/prem2017.h5`; their descriptor has a `loader` that reads whichever store has the location. `LAYOUTS` maps a layout name to the function that turns a stored array into a dataframe. ```{python} sorted(ssd.registry.LAYOUTS) ``` Adding a new family of data is a shaper plus a registry entry. ## How loading is dispatched ``` load(location, which, kind, revision, level) │ ├─ registry.dataset(which) → the descriptor, or an error with suggestions ├─ descriptor.loader? → delegate (e.g. `prem`, which picks prem2021 or prem2017) ├─ descriptor.tier > 0? → refuse, and point at fetch() or download() ├─ descriptor.source(revision) → which file, and which group inside it ├─ h5py.File(...) → open the store ├─ resolve(location, valid=...) → the canonical key, scoped to this dataset ├─ registry.get_layout(spec.layout) → the shaper for this layout └─ shaper(group, key, spec, opts) → the dataframe, plus provenance in .attrs ``` The `opts` dictionary carries the arguments that apply to some layouts and not others — `kind` for contact matrices, `level` for the node table — so a new layout can introduce its own without changing any existing shaper's signature. An argument applied to a layout that has no use for it is rejected rather than ignored: ```{python} #| error: true ssd.load('Kenya', which='deaths', level=1) ``` ## Storage Bundled data is stored as HDF5, one store per family, with a `spec` attribute describing the layout and a `provenance` attribute describing the source. The stores are self-describing: the loader reads how to interpret a file from the file itself. ```{python} #| echo: false pd.DataFrame([ dict(file='prem2021.h5, prem/prem2017.h5 (and, fetched, prem2017.h5, mistry2021.h5, mistry2021_us.h5)', layout='matrix', structure='// → a square float32 array'), dict(file='fetched.json', layout='—', structure='the stores that are fetched: URL, SHA-256, locations and provenance'), dict(file='demographics-.h5', layout='crude, mortality, population, fertility', structure='// → a (year[, sex][, age]) array'), dict(file='population_us.h5 (fetched)', layout='population', structure='/ → a (year, sex, age) array'), dict(file='nodes.h5', layout='nodes', structure='/ → a compound array, one row per unit'), dict(file='locations.csv', layout='—', structure='the canonical location table, read with the csv module'), ]) ``` Demographic stores are named by revision, so a new UN release adds a file rather than replacing one. The registry gathers the revisions of a series into a single descriptor, and `Dataset.source(revision)` selects among them, defaulting to the most recent. ## Tiers, in code Internally, whether a dataset is bundled is its *tier*, a field on the descriptor: 0 for bundled, 1 for fetched and 2 for downloaded (the descriptor's `access` property gives the same in the words used in [Bundled, fetched and downloaded data](access.qmd), and its `function` property the function for each). `load()` enforces it: ```{python} #| error: true ssd.load('kenya', which='life_table') ``` Tier-1 datasets appear in the registry despite having no bundled store, which is what lets `load()` redirect to `fetch()` or `download()` by name rather than fail to find a file. ## Extension points Three public hooks are provided, all of which are used internally by the package itself. **A new layout.** Register a function that turns a stored array into a dataframe: ```python def shape_my_layout(group, key, spec, opts): return sc.dataframe(group[key][:]) ssd.register_layout('my_layout', shape_my_layout) ``` **A new dataset.** Register a descriptor. Giving it a `loader` causes `load()` to delegate to it, which is the route for data this package does not ship: ```{python} ds = ssd.registry.Dataset(name='my_dataset', layout='crude', tier=2, description='Something from elsewhere', loader=lambda location, **kw: f'data for {location}') ssd.register_dataset(ds) ssd.load('Kenya', which='my_dataset') ``` ```{python} #| echo: false ssd.registry.unregister_dataset('my_dataset') ``` **A new plotter.** `plotting.PLOTTERS` maps a layout to a drawing function, and can be assigned into. All three hooks are functions rather than dictionaries you mutate, so a downstream package can extend the registry without reaching into it. ## Configuration and side effects Importing the package creates no directories, opens no files and emits no warnings. Settings are read from the environment at import, but acted on only when a function needs them: the cache directory is resolved at call time and created when something is first written to it. ```{python} ssd.options.help() ``` There is no logging configuration; handler setup is left to the application. ## Testing The test suite is divided by what it requires. ```{python} #| echo: false pd.DataFrame([ dict(marker='(none)', requires='Nothing but the bundled data', note='The majority; hermetic and fast'), dict(marker='network', requires='Internet access', note='Deselected by default in CI; set STARSIM_DATA_CACHE_DIR to reuse downloads'), dict(marker='slow', requires='More than a second or so', note='Chiefly the 145 MB life-table downloads'), dict(marker='spatial', requires='The [spatial] extra', note='Skipped automatically when it is absent'), ]) ``` Continuous integration also rebuilds every store from `raw_data/` and re-runs the suite against the result, so the committed artifacts cannot drift from their sources. The node table has a separate, manually triggered workflow, since rebuilding it downloads roughly 1.4 GB. ### Provenance and reproducibility The data in this package comes from other organizations, under several different licenses, and the numbers a model produces depend on which version of which source it used. This page shows how to find out where a dataset came from, how to cite it, and how to make an analysis reproducible. ## Attribution travels with the data Every dataset records its source, URL, license and citation, along with when and by which package version it was built. `provenance()` shows it, for fetched datasets too, without downloading anything: ```{python} import starsim_data as ssd ssd.provenance('prem2021') ``` The same record is attached to every dataframe `load()` returns: ```{python} deaths = ssd.load('kenya', which='deaths') deaths.attrs['provenance'].citation ``` ```{python} deaths.attrs['location'], deaths.attrs['dataset'] ``` Provenance for every dataset is available at once: ```{python} sorted(ssd.provenance().keys()) ``` ## Licenses The data is not covered by this package's MIT license; each dataset keeps its original license. ```{python} ssd.list_datasets(detailed='full')[['dataset', 'access', 'license']] ``` [NOTICE.md](https://github.com/starsimhub/starsim_data/blob/main/NOTICE.md) has the full attribution text, and the `licenses/` folder in the repository has the original license texts. GADM boundaries are licensed for non-commercial use only. This package never uses GADM unless you ask for it by name, and never redistributes anything derived from it; if you download it, following its terms is up to you. See [Data sources](datasources.qmd#gadm). ## Downloaded data Data downloaded from its original publisher (with `download()` and the `download_*()` functions) is saved on your computer, along with a note of where and when it was downloaded. `ssd.cache_info()` lists everything saved so far: ```python ssd.cache_info() # file size url downloaded # wpp/WPP2024_Demographic_Indicators_Medium.csv.gz 16.557 MB https://population.un.org/... 2026-08-07T19:31:36+00:00 ``` Dataframes returned by `fetch()` and `download()` carry the same information in `df.attrs['provenance']`. ## Making an analysis reproducible Four steps, most important first. **Pin the package version.** Everything bundled or fetched is the same for everyone using a given version of the package, so pinning the version (for example `starsim_data==2.2.0` in your requirements file) fixes the demographics, the contact matrices and the node table together. **Pin the WPP revision.** `load(..., revision=2024)` fixes the UN WPP release. Without it, `load()` uses the most recent revision in the package, so a future version could change what your script reads. Older revisions are kept when new ones are added, so a pinned revision stays available. **Record what was downloaded.** Data downloaded from its original publisher depends on when you downloaded it, since publishers revise their files. Saving the output of `ssd.cache_info()` with your results records what you used and when. **Record the boundary source.** Different boundary sources don't always divide a country up the same way, so two analyses that used different sources aren't directly comparable. The source is in `nodes.source` and in the provenance of any downloaded boundaries. If you're curious how the bundled data is built from its sources, and how that is checked, see [Building the data](building-the-data.qmd). ## Citation Cite the original sources, not just this package: UN World Population Prospects 2024 for the demographics, the relevant paper for each contact-matrix source you use, and UNOCHA and geoBoundaries for the node table. The citations are available in Python: ```{python} {name: ssd.provenance(name).citation[:80] + '...' for name in ['deaths', 'prem2021', 'mistry2021']} ``` [NOTICE.md](https://github.com/starsimhub/starsim_data/blob/main/NOTICE.md) gives the complete text for each source. ### Building the data The `.h5` files inside the package are build artifacts, generated from human-readable sources by the scripts in `scripts/`. This page describes what each script does and how to regenerate or update the data. It is for the package's maintainers: users never need to run these scripts, since every release ships its data ready to use. It is relevant when a new upstream release appears, when a source needs correcting, or when adding a dataset. ## Layout of the repository ``` starsim_data/ ├── raw_data/ # human-readable sources, tracked in git │ ├── contact_matrices/ # per-country CSVs, one folder per source │ └── demographics/ # three normalized UN WPP CSVs ├── scripts/ # build tools; not part of the installed package │ ├── download_wpp.py # raw_data/demographics/ ← UN WPP Data Portal API │ ├── build_hdf5.py # prem2021.h5, prem/prem2017.h5, fetched.json (and fetched/*.h5 with --fetched) │ ├── build_demographics.py # demographics-.h5 │ ├── build_locations.py # the derived columns of locations.csv │ ├── build_nodes.py # nodes.h5 │ └── provenance.py # shared: stamps each store with its source and build ├── fetched/ # stores fetched on demand: committed, not shipped └── starsim_data/data/ # the build artifacts, committed and shipped ``` Everything in `scripts/` requires `pip install -e ".[dev]"`, and `build_nodes.py` additionally requires `".[spatial]"`. Each script takes a `--check` flag, which builds into a temporary folder, compares the result with the committed files, and reads them back through the loader, without writing anything. It fails if the committed data differs from what the raw data builds, which is what CI runs to keep the two in step. Run the script without `--check` to write the files, then with it to confirm. ## Contact matrices ```bash python scripts/build_hdf5.py # build python scripts/build_hdf5.py --check # confirm the committed stores match a rebuild ``` Reads the per-location CSVs in `raw_data/contact_matrices/` and writes one store per source, keyed by location, with one `float32` matrix per contact setting. The source CSVs were derived from [epydemix-data](https://github.com/epistorm/epydemix-data), trimmed to national level and reorganized into a flat layout. The US states' were added later by `scripts/download_epydemix.py`, from a pinned commit of epydemix-data, as folders named like `United_States__Massachusetts` (the `__` stands for the `:` of the location key), along with their populations in `raw_data/population_us.csv`, which this script also builds into `population_us.h5`. Attribution for each source is held in a `SOURCES` table in the script, and the script warns if a source has no attribution recorded, so that adding a dataset without a license is noticed at build time rather than at review. `prem2021` is written into the package, together with the `prem2017` matrices for the ten locations `prem2021` lacks (`prem/prem2017.h5`), which the default contact matrices use. The full `prem2017` and `mistry2021`, and the US states' `mistry2021_us` and `population_us`, are kept in the repository's `fetched/` folder instead, outside the package, and published as assets of a GitHub release, which is where `fetch()` downloads them from. The package records exactly which version of each file it expects, so a given release always reads the same data. These matrices are already published and won't change, so the script leaves the fetched files alone. Its `--fetched` option rebuilds them (or only those named, as in `--fetched population_us`), which is only needed to add a new fetched source: its file is downloaded from a GitHub release, as an asset, so adding one means a new data release, named in the script's `RELEASE`. ## Demographics Two steps. The first retrieves the data: ```bash python scripts/download_wpp.py ``` This queries the UN WPP Data Portal API for the indicators the package uses, filters to decadal years and to the package's countries, and writes four normalized CSVs into `raw_data/demographics/`. It requires a free Bearer token in `scripts/.env`, which is git-ignored; the API's metadata endpoints are open but its data endpoints are not. The CSVs are tracked in git, so the second step needs no token and no network: ```bash python scripts/build_demographics.py ``` This classifies each CSV by its columns rather than by its filename, so bulk files downloaded manually from the WPP website work equally well. It writes `demographics-.h5` with one group per series. The script warns about any location present in `locations.csv` that ends up with no data, which catches silent coverage loss. ### Adding a new WPP revision ```bash python scripts/download_wpp.py python scripts/build_demographics.py --revision 2026 ``` This writes a new store alongside the existing one rather than replacing it. `load()` then uses the new revision by default, and `load(..., revision=2024)` continues to read the old one. Existing analyses that pinned a revision are unaffected. ## Locations ```bash python scripts/build_locations.py --check ``` `locations.csv` is partly hand-maintained and partly generated. The `location`, `level`, `iso_code`, `primary_contact_source`, `contact_sources` and `population_source` columns are edited by hand; `iso3`, `official_name` and `aliases` are filled in from `pycountry`, joining on the hand-maintained ISO-2 code. Because the generated columns are baked into the CSV at build time, `pycountry` is a build-time dependency only, and the resolver at runtime reads the file and needs nothing else. Aliases are drawn from `pycountry`'s official and common names, its French, Spanish and Portuguese translations, and a hand-curated table of abbreviations and former names that no automatic source provides — `USA`, `DRC`, `UK`, `Zaire` and similar. Any alias that would be ambiguous between two locations is discarded, with a warning. ## The node table ```bash python scripts/build_nodes.py # roughly 1.4 GB on a cold cache python scripts/build_nodes.py --check # confirm the committed stores match a rebuild python scripts/build_nodes.py --limit 5 # a subset, for development ``` For each country, the script prefers UNOCHA boundaries where the mirror covers it and falls back to the two global geoBoundaries CGAZ files otherwise. It computes an equal-area centroid and a geodesic area for every unit, assigns identifiers, and reads national population from the package's own bundled WPP data. Downloads pass through the package cache, so a rebuild after the first is quick. `--limit` is useful while developing, since a full build reads several hundred megabytes of geometry. The script refuses to draw on any source whose descriptor sets `redistributable=False`, which is how GADM's non-commercial license is kept out of bundled data. The check is performed against the same descriptors `download_shapes()` uses, so the two cannot disagree. For level-1 units, the script also sums WorldPop's 2020 raster within each unit for its population (about 1 GB of rasters, from a slow server), and works out which units share a border, which it writes to `adjacency.h5`. Other years and finer levels are left to `download_nodes()`. ## Adding a dataset Roughly the sequence to follow: 1. Put the human-readable source in `raw_data/`, if it is small enough to track. If it is not, it should be fetched or downloaded rather than bundled. 2. Write a build script that produces an `.h5` store with a `spec` attribute naming its layout, and a `provenance` attribute produced by `scripts/provenance.py`. 3. If the shape is unlike anything already present, add a layout shaper and register it; otherwise reuse an existing layout. See [Architecture](architecture.qmd). 4. Add a short description (about 35 characters, so `list_datasets(detailed=True)` fits on screen) to `registry.DESCRIPTIONS`. 5. Add the source, its license and its citation to `NOTICE.md`, and the license text to `licenses/` if the license requires it. 6. Add tests. At minimum: that the data loads, that its shape and units are as documented, and that its recorded license matches `NOTICE.md`. 7. Add the build step to the reproducibility job in `.github/workflows/tests.yml`, unless the inputs are too large to download on every run. ## Verifying a build ```bash pytest tests -m "not network" # the offline suite pytest tests # including downloads ``` The tests that concern build correctness are `test_demographics.py`, which exercises the whole demographic pipeline end to end using small synthetic WPP-format CSVs, and `test_provenance.py`, which asserts that every bundled dataset is attributable and that its recorded license matches `NOTICE.md`. Beyond the tests, plotting the result is a quick check on whether the numbers are plausible: ```python ssd.plot('kenya') ``` A units error, a wrong country or a misaligned age axis is usually obvious in a figure and easy to miss in a table. ### Changelog ```{python} #| echo: false #| output: asis # CHANGELOG.md, without its own "# Changelog" heading, which the page title already gives. # Pages run in docs/_quarto/default/user-guide/, the copy great-docs builds from. import pathlib text = pathlib.Path('../../../../CHANGELOG.md').read_text(encoding='utf-8') print(text.split('\n', 1)[1]) ```