# Data sources

🧠5% 🤖95%

The sources this package uses cover different sets of countries, are updated on different schedules, come under different licenses, and sometimes disagree with one another. This page compares them: what each is good for, its limitations, and how to choose between them. Much of the material on boundaries and population grids is adapted from [`laser-init`](https://github.com/laser-base/laser-init); see [Related projects](related-projects.md).

The full attribution and license text is in [NOTICE.md](https://github.com/starsimhub/starsim_data/blob/main/NOTICE.md). You can also get each dataset's source, license and citation in Python with `ssd.provenance(which)`, or from `df.attrs['provenance']` on any dataframe the package returns; see [Provenance and reproducibility](provenance.md).

## What comes from where

| Data | Source | Access | How you get it |
|----|----|----|----|
| Births, deaths, migration, population | UN WPP 2024 | bundled | [load()](../api/loaders.md#starsim_data.loaders.load) |
| Contact matrices | Prem 2017 / Prem 2021 / Mistry 2021 | bundled (`prem`), fetched (the rest) | [load()](../api/loaders.md#starsim_data.loaders.load) |
| Node table: centroids, areas, populations, p-codes | UNOCHA COD-AB, geoBoundaries | bundled | [load_nodes()](../api/nodes.md#starsim_data.nodes.load_nodes) |
| Annual series, life tables, other variants | UN WPP 2024 bulk files | downloaded | [download()](../api/downloading.md#starsim_data.downloading.download) |
| Administrative boundaries with geometry | UNOCHA, geoBoundaries, GADM | downloaded | [download_shapes()](../api/shapes.md#starsim_data.shapes.download_shapes) |
| Gridded population | WorldPop | downloaded | [download_raster()](../api/rasters.md#starsim_data.rasters.download_raster), [download_nodes()](../api/rasters.md#starsim_data.rasters.download_nodes) |

Bundled data is inside the package, works offline, and is the same for everyone using the same version. Fetched data is published with this package's releases and downloaded the first time you use it. Downloaded data comes straight from its original publisher. Fetched and downloaded files are saved on your computer, so each is downloaded only once. What goes where depends mostly on size, which is why some UN WPP data is bundled and the rest is downloaded; see [Bundled, fetched and downloaded data](access.md).

------------------------------------------------------------------------

## Demographics

### UN World Population Prospects

**Provider:** UN Department of Economic and Social Affairs, Population Division · [population.un.org/wpp](https://population.un.org/wpp/) · CC BY 3.0 IGO

The standard national demographic series, covering all countries from 1950 to 2100 and revised every two years.

|  |  |
|----|----|
| **Coverage** | 187 countries (regional totals are left out) |
| **Levels** | National only |
| **Variants** | Medium (bundled), plus High, Low, Constant fertility and others via [download()](../api/downloading.md#starsim_data.downloading.download) |
| **Update** | Every two years: 2022, 2024, … |

**Strengths.** The estimates use the same methods for every country, so countries can be compared, and the time series is long and consistent. The methods are well documented, and the data is revised often enough to stay current.

**Limitations.** The data is national only, so you can't get province-level rates from it. Projections are scenarios rather than predictions. Estimates can change a lot between revisions, since a new census can revise a country's past figures too, which is why you can pick a revision with `load(..., revision=2024)`.

**Two ways to get it.** The bundled data, which [load()](../api/loaders.md#starsim_data.loaders.load) reads, covers every tenth year and is small enough to ship with the package. [download()](../api/downloading.md#starsim_data.downloading.download) reads the UN's own files instead, for annual data, complete life tables and other projection variants; each file is a 17-290 MB download, needed once.

**If you read WPP files yourself:** Namibia's two-letter code is `NA`, which pandas reads as a missing value unless you pass `keep_default_na=False`. This package handles it for you.

------------------------------------------------------------------------

## Contact matrices

Three sources are available. `prem2021` is bundled; the other two are fetched the first time you use them. They differ mainly in age groups and country coverage. The default contact matrices, `prem`, are `prem2021`, filled in with `prem2017` for the ten locations `prem2021` lacks; see [Datasets](datasets.md#which-source).

| Source | Access | Matrix | Age bins | Countries | License |
|----|----|----|----|----|----|
| `prem2021` | bundled | 16×16 | 5-year, 0-75+ | 177 | GPL-3.0 |
| `prem2017` | fetched | 16×16 | 5-year, 0-75+ | 152 | GPL-3.0 |
| `mistry2021` | fetched | 85×85 | single-year, 0-84+ | 36 | Apache-2.0 |
| `mistry2021_us` | fetched | 85×85 | single-year, 0-84+ | 51 US states (with DC) | Apache-2.0 |

**Prem et al. 2017 and 2021** estimate contact patterns for countries without their own contact survey, based on household composition and age structure. The 2021 paper is an update that covers more countries and compares the results with survey data; use it unless you are reproducing earlier work. Both provide the same five settings: `all`, `home`, `work`, `school` and `community`.

**Mistry et al. 2021** has single-year age groups up to 84, but for only 36 countries. Use it when five-year age groups are too coarse, for example for school-entry effects, and your country is covered. The same study's matrices for the US states are `mistry2021_us`; each reflects that state's own households, schools and workplaces rather than the national average.

Not every country is in every source, so check with `list_locations('prem2021')` (or the source you want). The matrices were adapted from [epydemix-data](https://github.com/epistorm/epydemix-data), which also has subnational matrices for other countries and US counties that aren't included here; see [Related projects](related-projects.md).

### US Census Bureau

**Provider:** US Census Bureau, Population Estimates Program · [census.gov/popest](https://www.census.gov/programs-surveys/popest.html) · public domain

`population_us` is the Vintage 2023 estimate of each state's population by single year of age, as compiled by epydemix-data. It covers the 50 states and the District of Columbia (not the territories), for 2023 only, both sexes together, with the last age group 84+ to match `mistry2021_us`. Population totals for other years, or by sex, are not included.

------------------------------------------------------------------------

## Administrative boundaries

The three boundary sources complement each other, which is why all three are available and why [download_shapes()](../api/shapes.md#starsim_data.shapes.download_shapes) tries more than one.

### UNOCHA COD-AB

**Provider:** UN Office for the Coordination of Humanitarian Affairs · downloaded from [laser-base's copy](https://github.com/laser-base/unocha) · CC BY 3.0 IGO

|  |  |
|----|----|
| **Coverage** | ~110 countries, weighted towards humanitarian operations |
| **Levels** | 0-4, varying by country |
| **Download size** | 0.2-5 MB per country |
| **Key** | **P-code** |

**Strengths.** The p-codes are the reason this source is tried first. A p-code is the standard code the UN uses for each subnational area, and most humanitarian and health data uses it too, including surveillance line lists, campaign coverage and datasets on [HDX](https://data.humdata.org), so you can join that data to these boundaries by code. The boundaries are usually the official government ones, checked through UN operations, and are updated quickly during an emergency response.

**Limitations.** It covers about 110 countries, mostly those with humanitarian operations; many middle- and high-income countries are missing.

### geoBoundaries

**Provider:** William & Mary geoLab · [geoboundaries.org](https://www.geoboundaries.org) · CC BY 4.0

|                   |                                 |
|-------------------|---------------------------------|
| **Coverage**      | Near-universal (~200 countries) |
| **Levels**        | 0-5, varying by country         |
| **Download size** | 1-50 MB per country             |
| **Key**           | `shapeID` (not a p-code)        |

**Strengths.** It covers the countries UNOCHA doesn't. It is published in numbered releases, and this package uses a fixed one (v6.0.0), so the boundaries don't change from one run to the next. The methods are peer-reviewed and documented.

**Limitations.** Quality varies more by country and level than GADM's does. It uses its own IDs rather than p-codes, so joining p-coded data needs a lookup table between the two. Recent boundary changes may not be included yet.

### GADM

**Provider:** Global Administrative Areas · [gadm.org](https://gadm.org) · **non-commercial use only**

|                   |                                               |
|-------------------|-----------------------------------------------|
| **Coverage**      | Universal                                     |
| **Levels**        | 0-5, the deepest hierarchies of the three     |
| **Download size** | 1-100 MB per country (all levels in one file) |
| **Key**           | `GID_n`                                       |

**Strengths.** The most complete and most detailed of the three, with names in several languages and frequent updates.

**The license is the catch.** GADM is free for non-commercial use only, which rules it out for much applied public-health work. So this package:

- never uses GADM automatically: you have to ask for it with `download_shapes(..., source='gadm')`, which prints a reminder of the terms;
- never redistributes anything derived from it: the bundled node table is not built from GADM.

If you download GADM data, following its terms is up to you.

### Choosing between them

With no [source](../api/registry.md#starsim_data.registry.Dataset.source) argument, [download_shapes()](../api/shapes.md#starsim_data.shapes.download_shapes) tries UNOCHA and then geoBoundaries. That suits most disease modeling: p-codes where they exist, and coverage everywhere else. Choose a source yourself when:

- **you need level 4-5** -- geoBoundaries or GADM;
- **you are reproducing a published analysis** -- whichever it used;
- **you need a country's official boundaries** -- check all three against the national statistical office, since they do disagree;
- **you are joining to p-coded data** -- UNOCHA; if UNOCHA lacks the country, you'll need a lookup table between p-codes and the other source's IDs.

One caution: the sources don't always agree on how many units a country has, or where the boundaries are. Using different sources for different countries is usually fine; switching sources for the same country over time usually is not. The node table records the [source](../api/registry.md#starsim_data.registry.Dataset.source) of each row for this reason.

------------------------------------------------------------------------

## Gridded population

### WorldPop

**Provider:** WorldPop, University of Southampton · [worldpop.org](https://www.worldpop.org) · CC BY 4.0

|  |  |
|----|----|
| **Coverage** | Global, with gaps for very small territories |
| **Resolution** | 1 km (this package); 100 m available upstream |
| **Years** | 2000-2030 across two product series |
| **Download size** | 1-40 MB per country and year |
| **Method** | Census counts spread over a grid using a statistical model (satellite imagery, settlements, roads and so on) |

WorldPop estimates how many people live in each 1 km square. Adding those up within each administrative unit gives a population for each node, which [download_nodes()](../api/rasters.md#starsim_data.rasters.download_nodes) does for you.

**Strengths.** Global, open, detailed, adjusted to UN totals, and widely validated. Because there is a series of years, a model's population distribution can change over time rather than staying fixed.

**Limitations.** These are modeled estimates, not census counts. They don't add up exactly to the UN WPP national total for the same year; for Kenya in 2020 the WorldPop total is about 1.2% lower. Neither figure is wrong, as they estimate slightly different things. If your model needs the national total to match WPP, use [rescale_to_national()](../api/rasters.md#starsim_data.rasters.rescale_to_national). Estimates are also more accurate in cities than in rural areas, and come without uncertainty ranges.

WorldPop's server is slow, so the first download for a country can take a while; after that it is saved on your computer and reused.

------------------------------------------------------------------------

## Reproducibility

The original publishers revise their data over time. To keep an analysis reproducible, pin the `starsim_data` version, pick the WPP revision explicitly, and record the boundary source you used. [Provenance and reproducibility](provenance.md) has the details.

------------------------------------------------------------------------

## Not included

These aren't included; here is where else to look:

- **Subnational demographics.** WPP provides none. National statistical offices, DHS and IPUMS are the usual alternatives.
- **Age-specific migration.** WPP publishes the crude national rate only.
- **Other population grids** -- LandScan, GHSL, GRID3, Meta HRSL. WorldPop was chosen because it is open and globally consistent; the others are better for some questions.
- **OpenStreetMap and Natural Earth boundaries.** OpenStreetMap's quality varies a lot from place to place; Natural Earth is simplified for drawing maps rather than for analysis.
- **Health-system, mobility and climate data.** Out of scope for this package.
