# Building the data

🧠10% 🤖90%

The `.h5` files inside the package are build artifacts, generated from human-readable sources by the scripts in `scripts/`. This page describes what each script does and how to regenerate or update the data. It is for the package's maintainers: users never need to run these scripts, since every release ships its data ready to use. It is relevant when a new upstream release appears, when a source needs correcting, or when adding a dataset.

## Layout of the repository

    starsim_data/
    ├── raw_data/                   # human-readable sources, tracked in git
    │   ├── contact_matrices/       # per-country CSVs, one folder per source
    │   └── demographics/           # three normalized UN WPP CSVs
    ├── scripts/                    # build tools; not part of the installed package
    │   ├── download_wpp.py         # raw_data/demographics/  ← UN WPP Data Portal API
    │   ├── build_hdf5.py           # prem2021.h5, prem/prem2017.h5, fetched.json (and fetched/*.h5 with --fetched)
    │   ├── build_demographics.py   # demographics-<revision>.h5
    │   ├── build_locations.py      # the derived columns of locations.csv
    │   ├── build_nodes.py          # nodes.h5
    │   └── provenance.py           # shared: stamps each store with its source and build
    ├── fetched/                    # stores fetched on demand: committed, not shipped
    └── starsim_data/data/          # the build artifacts, committed and shipped

Everything in `scripts/` requires `pip install -e ".[dev]"`, and `build_nodes.py` additionally requires `".[spatial]"`.

Each script takes a `--check` flag, which builds into a temporary folder, compares the result with the committed files, and reads them back through the loader, without writing anything. It fails if the committed data differs from what the raw data builds, which is what CI runs to keep the two in step. Run the script without `--check` to write the files, then with it to confirm.

## Contact matrices

``` bash
python scripts/build_hdf5.py           # build
python scripts/build_hdf5.py --check   # confirm the committed stores match a rebuild
```

Reads the per-location CSVs in `raw_data/contact_matrices/` and writes one store per source, keyed by location, with one `float32` matrix per contact setting. The source CSVs were derived from [epydemix-data](https://github.com/epistorm/epydemix-data), trimmed to national level and reorganized into a flat layout. The US states' were added later by `scripts/download_epydemix.py`, from a pinned commit of epydemix-data, as folders named like `United_States__Massachusetts` (the `__` stands for the `:` of the location key), along with their populations in `raw_data/population_us.csv`, which this script also builds into `population_us.h5`.

Attribution for each source is held in a `SOURCES` table in the script, and the script warns if a source has no attribution recorded, so that adding a dataset without a license is noticed at build time rather than at review.

`prem2021` is written into the package, together with the `prem2017` matrices for the ten locations `prem2021` lacks (`prem/prem2017.h5`), which the default contact matrices use. The full `prem2017` and `mistry2021`, and the US states' `mistry2021_us` and `population_us`, are kept in the repository's `fetched/` folder instead, outside the package, and published as assets of a GitHub release, which is where [fetch()](../api/fetching.md#starsim_data.fetching.fetch) downloads them from. The package records exactly which version of each file it expects, so a given release always reads the same data.

These matrices are already published and won't change, so the script leaves the fetched files alone. Its `--fetched` option rebuilds them (or only those named, as in `--fetched population_us`), which is only needed to add a new fetched source: its file is downloaded from a GitHub release, as an asset, so adding one means a new data release, named in the script's `RELEASE`.

## Demographics

Two steps. The first retrieves the data:

``` bash
python scripts/download_wpp.py
```

This queries the UN WPP Data Portal API for the indicators the package uses, filters to decadal years and to the package's countries, and writes four normalized CSVs into `raw_data/demographics/`. It requires a free Bearer token in `scripts/.env`, which is git-ignored; the API's metadata endpoints are open but its data endpoints are not.

The CSVs are tracked in git, so the second step needs no token and no network:

``` bash
python scripts/build_demographics.py
```

This classifies each CSV by its columns rather than by its filename, so bulk files downloaded manually from the WPP website work equally well. It writes `demographics-<revision>.h5` with one group per series.

The script warns about any location present in `locations.csv` that ends up with no data, which catches silent coverage loss.

### Adding a new WPP revision

``` bash
python scripts/download_wpp.py
python scripts/build_demographics.py --revision 2026
```

This writes a new store alongside the existing one rather than replacing it. [load()](../api/loaders.md#starsim_data.loaders.load) then uses the new revision by default, and `load(..., revision=2024)` continues to read the old one. Existing analyses that pinned a revision are unaffected.

## Locations

``` bash
python scripts/build_locations.py --check
```

`locations.csv` is partly hand-maintained and partly generated. The `location`, [level](../api/registry.md#starsim_data.registry.Dataset.level), `iso_code`, `primary_contact_source`, `contact_sources` and `population_source` columns are edited by hand; `iso3`, `official_name` and `aliases` are filled in from `pycountry`, joining on the hand-maintained ISO-2 code.

Because the generated columns are baked into the CSV at build time, `pycountry` is a build-time dependency only, and the resolver at runtime reads the file and needs nothing else.

Aliases are drawn from `pycountry`'s official and common names, its French, Spanish and Portuguese translations, and a hand-curated table of abbreviations and former names that no automatic source provides -- `USA`, `DRC`, `UK`, `Zaire` and similar. Any alias that would be ambiguous between two locations is discarded, with a warning.

## The node table

``` bash
python scripts/build_nodes.py             # roughly 1.4 GB on a cold cache
python scripts/build_nodes.py --check     # confirm the committed stores match a rebuild
python scripts/build_nodes.py --limit 5   # a subset, for development
```

For each country, the script prefers UNOCHA boundaries where the mirror covers it and falls back to the two global geoBoundaries CGAZ files otherwise. It computes an equal-area centroid and a geodesic area for every unit, assigns identifiers, and reads national population from the package's own bundled WPP data.

Downloads pass through the package cache, so a rebuild after the first is quick. `--limit` is useful while developing, since a full build reads several hundred megabytes of geometry.

The script refuses to draw on any source whose descriptor sets `redistributable=False`, which is how GADM's non-commercial license is kept out of bundled data. The check is performed against the same descriptors [download_shapes()](../api/shapes.md#starsim_data.shapes.download_shapes) uses, so the two cannot disagree.

For level-1 units, the script also sums WorldPop's 2020 raster within each unit for its population (about 1 GB of rasters, from a slow server), and works out which units share a border, which it writes to `adjacency.h5`. Other years and finer levels are left to [download_nodes()](../api/rasters.md#starsim_data.rasters.download_nodes).

## Adding a dataset

Roughly the sequence to follow:

1.  Put the human-readable source in `raw_data/`, if it is small enough to track. If it is not, it should be fetched or downloaded rather than bundled.
2.  Write a build script that produces an `.h5` store with a `spec` attribute naming its layout, and a [provenance](../api/loaders.md#starsim_data.loaders.provenance) attribute produced by `scripts/provenance.py`.
3.  If the shape is unlike anything already present, add a layout shaper and register it; otherwise reuse an existing layout. See [Architecture](architecture.md).
4.  Add a short description (about 35 characters, so `list_datasets(detailed=True)` fits on screen) to `registry.DESCRIPTIONS`.
5.  Add the source, its license and its citation to `NOTICE.md`, and the license text to `licenses/` if the license requires it.
6.  Add tests. At minimum: that the data loads, that its shape and units are as documented, and that its recorded license matches `NOTICE.md`.
7.  Add the build step to the reproducibility job in `.github/workflows/tests.yml`, unless the inputs are too large to download on every run.

## Verifying a build

``` bash
pytest tests -m "not network"     # the offline suite
pytest tests                      # including downloads
```

The tests that concern build correctness are `test_demographics.py`, which exercises the whole demographic pipeline end to end using small synthetic WPP-format CSVs, and `test_provenance.py`, which asserts that every bundled dataset is attributable and that its recorded license matches `NOTICE.md`.

Beyond the tests, plotting the result is a quick check on whether the numbers are plausible:

``` python
ssd.plot('kenya')
```

A units error, a wrong country or a misaligned age axis is usually obvious in a figure and easy to miss in a table.
