Building the data
The .h5 files inside the package are build artifacts, generated from human-readable sources by the scripts in scripts/. This page describes what each script does and how to regenerate or update the data. It is for the package’s maintainers: users never need to run these scripts, since every release ships its data ready to use. It is relevant when a new upstream release appears, when a source needs correcting, or when adding a dataset.
Layout of the repository
starsim_data/
├── raw_data/ # human-readable sources, tracked in git
│ ├── contact_matrices/ # per-country CSVs, one folder per source
│ └── demographics/ # three normalized UN WPP CSVs
├── scripts/ # build tools; not part of the installed package
│ ├── download_wpp.py # raw_data/demographics/ ← UN WPP Data Portal API
│ ├── build_hdf5.py # prem2021.h5, prem/prem2017.h5, fetched.json (and fetched/*.h5 with --fetched)
│ ├── build_demographics.py # demographics-<revision>.h5
│ ├── build_locations.py # the derived columns of locations.csv
│ ├── build_nodes.py # nodes.h5
│ └── provenance.py # shared: stamps each store with its source and build
├── fetched/ # stores fetched on demand: committed, not shipped
└── starsim_data/data/ # the build artifacts, committed and shipped
Everything in scripts/ requires pip install -e ".[dev]", and build_nodes.py additionally requires ".[spatial]".
Each script takes a --check flag, which builds into a temporary folder, compares the result with the committed files, and reads them back through the loader, without writing anything. It fails if the committed data differs from what the raw data builds, which is what CI runs to keep the two in step. Run the script without --check to write the files, then with it to confirm.
Contact matrices
python scripts/build_hdf5.py # build
python scripts/build_hdf5.py --check # confirm the committed stores match a rebuildReads the per-location CSVs in raw_data/contact_matrices/ and writes one store per source, keyed by location, with one float32 matrix per contact setting. The source CSVs were derived from epydemix-data, trimmed to national level and reorganized into a flat layout. The US states’ were added later by scripts/download_epydemix.py, from a pinned commit of epydemix-data, as folders named like United_States__Massachusetts (the __ stands for the : of the location key), along with their populations in raw_data/population_us.csv, which this script also builds into population_us.h5.
Attribution for each source is held in a SOURCES table in the script, and the script warns if a source has no attribution recorded, so that adding a dataset without a license is noticed at build time rather than at review.
prem2021 is written into the package, together with the prem2017 matrices for the ten locations prem2021 lacks (prem/prem2017.h5), which the default contact matrices use. The full prem2017 and mistry2021, and the US states’ mistry2021_us and population_us, are kept in the repository’s fetched/ folder instead, outside the package, and published as assets of a GitHub release, which is where fetch() downloads them from. The package records exactly which version of each file it expects, so a given release always reads the same data.
These matrices are already published and won’t change, so the script leaves the fetched files alone. Its --fetched option rebuilds them (or only those named, as in --fetched population_us), which is only needed to add a new fetched source: its file is downloaded from a GitHub release, as an asset, so adding one means a new data release, named in the script’s RELEASE.
Demographics
Two steps. The first retrieves the data:
python scripts/download_wpp.pyThis queries the UN WPP Data Portal API for the indicators the package uses, filters to decadal years and to the package’s countries, and writes four normalized CSVs into raw_data/demographics/. It requires a free Bearer token in scripts/.env, which is git-ignored; the API’s metadata endpoints are open but its data endpoints are not.
The CSVs are tracked in git, so the second step needs no token and no network:
python scripts/build_demographics.pyThis classifies each CSV by its columns rather than by its filename, so bulk files downloaded manually from the WPP website work equally well. It writes demographics-<revision>.h5 with one group per series.
The script warns about any location present in locations.csv that ends up with no data, which catches silent coverage loss.
Adding a new WPP revision
python scripts/download_wpp.py
python scripts/build_demographics.py --revision 2026This writes a new store alongside the existing one rather than replacing it. load() then uses the new revision by default, and load(..., revision=2024) continues to read the old one. Existing analyses that pinned a revision are unaffected.
Locations
python scripts/build_locations.py --checklocations.csv is partly hand-maintained and partly generated. The location, level, iso_code, primary_contact_source, contact_sources and population_source columns are edited by hand; iso3, official_name and aliases are filled in from pycountry, joining on the hand-maintained ISO-2 code.
Because the generated columns are baked into the CSV at build time, pycountry is a build-time dependency only, and the resolver at runtime reads the file and needs nothing else.
Aliases are drawn from pycountry’s official and common names, its French, Spanish and Portuguese translations, and a hand-curated table of abbreviations and former names that no automatic source provides — USA, DRC, UK, Zaire and similar. Any alias that would be ambiguous between two locations is discarded, with a warning.
The node table
python scripts/build_nodes.py # roughly 1.4 GB on a cold cache
python scripts/build_nodes.py --check # confirm the committed stores match a rebuild
python scripts/build_nodes.py --limit 5 # a subset, for developmentFor each country, the script prefers UNOCHA boundaries where the mirror covers it and falls back to the two global geoBoundaries CGAZ files otherwise. It computes an equal-area centroid and a geodesic area for every unit, assigns identifiers, and reads national population from the package’s own bundled WPP data.
Downloads pass through the package cache, so a rebuild after the first is quick. --limit is useful while developing, since a full build reads several hundred megabytes of geometry.
The script refuses to draw on any source whose descriptor sets redistributable=False, which is how GADM’s non-commercial license is kept out of bundled data. The check is performed against the same descriptors download_shapes() uses, so the two cannot disagree.
For level-1 units, the script also sums WorldPop’s 2020 raster within each unit for its population (about 1 GB of rasters, from a slow server), and works out which units share a border, which it writes to adjacency.h5. Other years and finer levels are left to download_nodes().
Adding a dataset
Roughly the sequence to follow:
- Put the human-readable source in
raw_data/, if it is small enough to track. If it is not, it should be fetched or downloaded rather than bundled. - Write a build script that produces an
.h5store with aspecattribute naming its layout, and a provenance attribute produced byscripts/provenance.py. - If the shape is unlike anything already present, add a layout shaper and register it; otherwise reuse an existing layout. See Architecture.
- Add a short description (about 35 characters, so
list_datasets(detailed=True)fits on screen) toregistry.DESCRIPTIONS. - Add the source, its license and its citation to
NOTICE.md, and the license text tolicenses/if the license requires it. - Add tests. At minimum: that the data loads, that its shape and units are as documented, and that its recorded license matches
NOTICE.md. - Add the build step to the reproducibility job in
.github/workflows/tests.yml, unless the inputs are too large to download on every run.
Verifying a build
pytest tests -m "not network" # the offline suite
pytest tests # including downloadsThe tests that concern build correctness are test_demographics.py, which exercises the whole demographic pipeline end to end using small synthetic WPP-format CSVs, and test_provenance.py, which asserts that every bundled dataset is attributable and that its recorded license matches NOTICE.md.
Beyond the tests, plotting the result is a quick check on whether the numbers are plausible:
ssd.plot('kenya')A units error, a wrong country or a misaligned age axis is usually obvious in a figure and easy to miss in a table.