| module | role | |
|---|---|---|
| 0 | settings.py | The `options` object: cache location and verbo... |
| 1 | registry.py | What datasets exist, how each is shaped, and t... |
| 2 | locations.py | Resolving location strings; administrative-lev... |
| 3 | loaders.py | Reading bundled stores: `load`, `list_datasets... |
| 4 | cache.py | The download cache and its provenance sidecars |
| 5 | fetching.py | Stores published with the releases: `fetch` |
| 6 | downloading.py | UN WPP bulk files: `download` |
| 7 | nodes.py | The node table and `distances` |
| 8 | shapes.py | Boundary sources and `download_shapes` (needs ... |
| 9 | rasters.py | WorldPop rasters and `download_nodes` (needs [... |
| 10 | plotting.py | Validation figures and the choropleth |
Architecture
This page describes how the package is organized internally: what the modules do, how a call to load() is dispatched, how data is stored on disk, and where the extension points are. It is intended for people modifying the package or adding data to it, rather than for people using it.
Modules
Import order matters, because two modules register themselves when they are imported: loaders adds its layout shapers and downloading adds its tier-2 datasets, both of which require registry to exist first. The order is fixed in __init__.py.
Import conventions
Modules follow Starsimβs convention: each imports the package under its own alias and reaches everything through it.
import starsim_data as ssd
def load_nodes(location, level=0):
return ssd.load(location, which='nodes', level=level)from imports are confined to __init__.py, where the public namespace is assembled. Within a module, ssd.load and ssd.registry.datasets() resolve at call time, so modules can refer to each other freely.
The registry
The registry holds two tables.
DATASETS maps a dataset name to a descriptor recording its layout, tier, administrative levels, license, available revisions, location count and where its store lives.
import starsim_data as ssd
ssd.registry.dataset('deaths')Dataset(name='deaths', layout='mortality', tier=0, levels=(0,), sources=(Source(store='demographics-2024.h5', group='deaths', revision='2024', url='', sha256=''),), license='CC BY 3.0 IGO', n_locations=187, description='Mortality by age and sex (UN WPP)')
Descriptors for bundled data are discovered once, by opening each .h5 file in the data directory and reading its root attributes. The result is cached; registry.refresh() discards the cache, which is needed only if the files change within a running process, such as after a rebuild in a test. The default contact matrices, prem, are described at the same time, from prem2021.h5 and the ten matrices in prem/prem2017.h5; their descriptor has a loader that reads whichever store has the location.
LAYOUTS maps a layout name to the function that turns a stored array into a dataframe.
sorted(ssd.registry.LAYOUTS)['adjacency',
'crude',
'fertility',
'matrix',
'mortality',
'nodes',
'population']
Adding a new family of data is a shaper plus a registry entry.
How loading is dispatched
load(location, which, kind, revision, level)
β
ββ registry.dataset(which) β the descriptor, or an error with suggestions
ββ descriptor.loader? β delegate (e.g. `prem`, which picks prem2021 or prem2017)
ββ descriptor.tier > 0? β refuse, and point at fetch() or download()
ββ descriptor.source(revision) β which file, and which group inside it
ββ h5py.File(...) β open the store
ββ resolve(location, valid=...) β the canonical key, scoped to this dataset
ββ registry.get_layout(spec.layout) β the shaper for this layout
ββ shaper(group, key, spec, opts) β the dataframe, plus provenance in .attrs
The opts dictionary carries the arguments that apply to some layouts and not others β kind for contact matrices, level for the node table β so a new layout can introduce its own without changing any existing shaperβs signature. An argument applied to a layout that has no use for it is rejected rather than ignored:
ssd.load('Kenya', which='deaths', level=1)--------------------------------------------------------------------------- ValueError Traceback (most recent call last) Cell In[4], line 1 ----> 1 ssd.load('Kenya', which='deaths', level=1) File ~/work/starsim_data/starsim_data/starsim_data/loaders.py:287, in load(location, which, kind, revision, level) 285 return ds.loader(location, kind=kind, revision=revision, level=level) 286 if (level is not None) and (ds.layout != 'nodes'): --> 287 raise ValueError(f'level={level!r} does not apply to {which!r}; only the node table is addressed by administrative level') 288 if ds.tier > 0: # `load` never touches the network; say so rather than quietly downloading 289 raise ValueError(f'{which!r} is not bundled; use {ds.function}({location!r}, which={which!r}) to download it') ValueError: level=1 does not apply to 'deaths'; only the node table is addressed by administrative level
Storage
Bundled data is stored as HDF5, one store per family, with a spec attribute describing the layout and a provenance attribute describing the source. The stores are self-describing: the loader reads how to interpret a file from the file itself.
| file | layout | structure | |
|---|---|---|---|
| 0 | prem2021.h5, prem/prem2017.h5 (and, fetched, p... | matrix | /<Location>/<setting> β a square float32 array |
| 1 | fetched.json | β | the stores that are fetched: URL, SHA-256, loc... |
| 2 | demographics-<revision>.h5 | crude, mortality, population, fertility | /<series>/<Location> β a (year[, sex][, age]) ... |
| 3 | population_us.h5 (fetched) | population | /<Country:Unit> β a (year, sex, age) array |
| 4 | nodes.h5 | nodes | /<Location> β a compound array, one row per unit |
| 5 | locations.csv | β | the canonical location table, read with the cs... |
Demographic stores are named by revision, so a new UN release adds a file rather than replacing one. The registry gathers the revisions of a series into a single descriptor, and Dataset.source(revision) selects among them, defaulting to the most recent.
Tiers, in code
Internally, whether a dataset is bundled is its tier, a field on the descriptor: 0 for bundled, 1 for fetched and 2 for downloaded (the descriptorβs access property gives the same in the words used in Bundled, fetched and downloaded data, and its function property the function for each). load() enforces it:
ssd.load('kenya', which='life_table')--------------------------------------------------------------------------- ValueError Traceback (most recent call last) Cell In[6], line 1 ----> 1 ssd.load('kenya', which='life_table') File ~/work/starsim_data/starsim_data/starsim_data/loaders.py:289, in load(location, which, kind, revision, level) 287 raise ValueError(f'level={level!r} does not apply to {which!r}; only the node table is addressed by administrative level') 288 if ds.tier > 0: # `load` never touches the network; say so rather than quietly downloading --> 289 raise ValueError(f'{which!r} is not bundled; use {ds.function}({location!r}, which={which!r}) to download it') 290 return _read(which, location, kind=kind, revision=revision, level=level) ValueError: 'life_table' is not bundled; use download('kenya', which='life_table') to download it
Tier-1 datasets appear in the registry despite having no bundled store, which is what lets load() redirect to fetch() or download() by name rather than fail to find a file.
Extension points
Three public hooks are provided, all of which are used internally by the package itself.
A new layout. Register a function that turns a stored array into a dataframe:
def shape_my_layout(group, key, spec, opts):
return sc.dataframe(group[key][:])
ssd.register_layout('my_layout', shape_my_layout)A new dataset. Register a descriptor. Giving it a loader causes load() to delegate to it, which is the route for data this package does not ship:
ds = ssd.registry.Dataset(name='my_dataset', layout='crude', tier=2,
description='Something from elsewhere',
loader=lambda location, **kw: f'data for {location}')
ssd.register_dataset(ds)
ssd.load('Kenya', which='my_dataset')'data for Kenya'
Dataset(name='my_dataset', layout='crude', tier=2, levels=(0,), sources=(), license='', n_locations=0, description='Something from elsewhere')
A new plotter. plotting.PLOTTERS maps a layout to a drawing function, and can be assigned into.
All three hooks are functions rather than dictionaries you mutate, so a downstream package can extend the registry without reaching into it.
Configuration and side effects
Importing the package creates no directories, opens no files and emits no warnings. Settings are read from the environment at import, but acted on only when a function needs them: the cache directory is resolved at call time and created when something is first written to it.
ssd.options.help() cache_dir = None # Where downloaded (non-bundled) data is cached. None means a platform-appropriate default; see cache_dir().
verbose = True # Whether to report progress while downloading.
pandas = False # Whether to return plain pandas DataFrames rather than Sciris dataframes.
There is no logging configuration; handler setup is left to the application.
Testing
The test suite is divided by what it requires.
| marker | requires | note | |
|---|---|---|---|
| 0 | (none) | Nothing but the bundled data | The majority; hermetic and fast |
| 1 | network | Internet access | Deselected by default in CI; set STARSIM_DATA_... |
| 2 | slow | More than a second or so | Chiefly the 145 MB life-table downloads |
| 3 | spatial | The [spatial] extra | Skipped automatically when it is absent |
Continuous integration also rebuilds every store from raw_data/ and re-runs the suite against the result, so the committed artifacts cannot drift from their sources. The node table has a separate, manually triggered workflow, since rebuilding it downloads roughly 1.4 GB.