# Datasets and their shapes

🧠15% 🤖85%

This page lists the columns and units of each dataset. The examples are run when this page is built, so they show what the current version actually returns.

``` python
import starsim_data as ssd

ssd.list_datasets(detailed='full').drop(columns='license')
```

|  | dataset | description | layout | access | levels | locations | revisions |
|----|----|----|----|----|----|----|----|
| 0 | adjacency | Admin units that share a border | adjacency | bundled | 1 | 184 |  |
| 1 | births | Crude birth rate (UN WPP) | crude | bundled | 0 | 187 | 2024 |
| 2 | deaths | Mortality by age and sex (UN WPP) | mortality | bundled | 0 | 187 | 2024 |
| 3 | fertility | Fertility by age of mother (UN WPP) | fertility | bundled | 0 | 187 | 2024 |
| 4 | indicators | All demographic indicators (UN WPP) | wide | downloaded | 0 | 0 |  |
| 5 | life_table | Abridged life tables (UN WPP) | long | downloaded | 0 | 0 |  |
| 6 | migration | Net migration rate (UN WPP) | crude | bundled | 0 | 187 | 2024 |
| 7 | mistry2021 | Contact matrices (Mistry 2021) | matrix | fetched | 0 | 36 |  |
| 8 | mistry2021_us | US contact matrices (Mistry 2021) | matrix | fetched | 1 | 51 |  |
| 9 | nodes | Admin units: centroids, areas, codes | nodes | bundled | 0, 1 | 184 |  |
| 10 | population | Population by age and sex (UN WPP) | population | bundled | 0 | 187 | 2024 |
| 11 | population_us | US population by age (Census) | population | fetched | 1 | 51 |  |
| 12 | prem | Contact matrices (Prem 2021 + 2017) | matrix | bundled | 0 | 187 |  |
| 13 | prem2017 | Contact matrices (Prem 2017) | matrix | fetched | 0 | 152 |  |
| 14 | prem2021 | Contact matrices (Prem 2021) | matrix | bundled | 0 | 177 |  |

`layout` is how the data is shaped (the sections below describe each), `levels` the administrative levels it covers (0 for countries, 1 for states or provinces), and [revisions](../api/registry.md#starsim_data.registry.Dataset.revisions) the WPP revisions it is available for. Licenses are left out here to save space; they are listed in [Provenance](provenance.md), and in full in `NOTICE.md`.

## Demographics

Five series come from UN World Population Prospects (WPP) 2024, Medium variant, with one year per decade from 1950 to 2100 (sixteen years in all). All 187 countries have all five series.

The column names are the UN's own, so the data can be passed to Starsim's demographic modules without renaming anything.

### Births and migration

Both are national crude rates, one value per year, not broken down by age. The UN publishes no age breakdown for migration; births by age of mother are the `fertility` series, below.

``` python
births = ssd.load('kenya', which='births')
births.head()
```

|     | Time | CBR       |
|-----|------|-----------|
| 0   | 1950 | 49.337002 |
| 1   | 1960 | 55.183998 |
| 2   | 1970 | 52.540001 |
| 3   | 1980 | 48.210999 |
| 4   | 1990 | 43.930000 |

|     | column | dtype   | units                                | note               |
|-----|--------|---------|--------------------------------------|--------------------|
| 0   | Time   | int64   | calendar year                        | decadal, 1950-2100 |
| 1   | CBR    | float32 | births per 1,000 population per year |                    |

The `migration` series has the same shape, with `CNMR` (crude net-migration rate per 1,000) in place of `CBR`. It is signed, with negative values indicating net emigration.

``` python
ssd.load('kenya', which='migration').head(3)
```

|     | Time | CNMR   |
|-----|------|--------|
| 0   | 1950 | 1.090  |
| 1   | 1960 | 0.277  |
| 2   | 1970 | -1.149 |

### Deaths

Age- and sex-specific mortality, in the usual abridged age groups (`0`, `1-4`, `5-9`, …, `100+`). Only Female and Male are included, not the UN's "Both sexes" rows, to match Starsim.

``` python
deaths = ssd.load('kenya', which='deaths')
deaths.head()
```

|     | Time | Sex    | AgeGrpStart | mx       |
|-----|------|--------|-------------|----------|
| 0   | 1950 | Female | 0           | 0.160465 |
| 1   | 1950 | Female | 1           | 0.055043 |
| 2   | 1950 | Female | 5           | 0.012113 |
| 3   | 1950 | Female | 10          | 0.006858 |
| 4   | 1950 | Female | 15          | 0.006908 |

|  | column | dtype | units | note |
|----|----|----|----|----|
| 0 | Time | int64 | calendar year |  |
| 1 | Sex | string | Female \| Male | no "Both sexes" |
| 2 | AgeGrpStart | int64 | years | start of the bin: 0, 1, 5, 10, ..., 100 |
| 3 | mx | float32 | deaths per person-year | the central death rate m(x) |

> **Note: Death rates are not probabilities**
>
> `mx` is the central death rate, deaths per person-year lived, not the probability of dying in the age group (`qx`). The two are similar where mortality is low and differ where it is high. The other life-table columns, including `qx`, are available through [download()](../api/downloading.md#starsim_data.downloading.download), as described below.

``` python
len(deaths), deaths.Time.nunique(), deaths.AgeGrpStart.nunique()
```

    (704, 16, 22)

That is sixteen years times two sexes times twenty-two age groups.

### Population

Population by 5-year age group and sex, in persons. (The UN publishes these in thousands; they are converted here.)

``` python
population = ssd.load('kenya', which='population')
population.head()
```

|     | Time | Sex    | AgeGrp | Value    |
|-----|------|--------|--------|----------|
| 0   | 1950 | Female | 0      | 546394.0 |
| 1   | 1950 | Female | 5      | 418403.5 |
| 2   | 1950 | Female | 10     | 390101.0 |
| 3   | 1950 | Female | 15     | 338169.0 |
| 4   | 1950 | Female | 20     | 258914.0 |

|     | column | dtype   | units          | note                             |
|-----|--------|---------|----------------|----------------------------------|
| 0   | Time   | int64   | calendar year  |                                  |
| 1   | Sex    | string  | Female \| Male |                                  |
| 2   | AgeGrp | int64   | years          | start of the bin: 0, 5, ..., 100 |
| 3   | Value  | float32 | persons        | absolute count, not thousands    |

The national total is a quick check on the units:

``` python
f"{population[population.Time == 2020].Value.sum() / 1e6:.1f}M"
```

    '52.2M'

### Fertility

Age-specific fertility rates, by 5-year age group of the mother (`10-14`, …, `50-54`). There is no `Sex` column, since the rates are per woman.

``` python
fertility = ssd.load('kenya', which='fertility')
fertility.head()
```

|     | Time | AgeGrpStart | ASFR       |
|-----|------|-------------|------------|
| 0   | 1950 | 10          | 23.204000  |
| 1   | 1950 | 15          | 162.516006 |
| 2   | 1950 | 20          | 280.062988 |
| 3   | 1950 | 25          | 302.570007 |
| 4   | 1950 | 30          | 293.190002 |

|  | column | dtype | units | note |
|----|----|----|----|----|
| 0 | Time | int64 | calendar year |  |
| 1 | AgeGrpStart | int64 | years | start of the bin: 10, 15, ..., 50 |
| 2 | ASFR | float32 | births per 1,000 women per year | the age-specific fertility rate |

Summed over the age groups, times five years per group and divided by 1,000, it gives the total fertility rate:

``` python
f"{5 * fertility[fertility.Time == 2020].ASFR.sum() / 1000:.2f}"
```

    '3.37'

### Revisions

The UN updates WPP every two years or so. Each release (revision) is kept separately, so a new one is added alongside the old rather than changing data already in use:

``` python
ssd.list_revisions()
```

    ['2024']

``` python
ssd.load('kenya', which='deaths')                 # newest bundled revision
ssd.load('kenya', which='deaths', revision=2024)  # pinned
```

Naming the revision is a good idea in any analysis you want to be able to reproduce later.

## Contact matrices

Contact matrices give the average number of daily contacts between people in each pair of age groups. Three sources are available, each with five contact settings, selected with the `kind` argument. One of them, `prem2021`, is bundled; the other two are fetched the first time you use them (see [Bundled, fetched and downloaded data](access.md)). The default, `prem`, is `prem2021`, or `prem2017` for the ten locations `prem2021` lacks (see [Which source](#which-source)).

``` python
matrix = ssd.load('kenya')   # prem2021
matrix.iloc[:6, :6].round(3)
```

|     | 0     | 5     | 10    | 15    | 20    | 25    |
|-----|-------|-------|-------|-------|-------|-------|
| 0   | 2.402 | 1.808 | 1.109 | 0.656 | 0.812 | 1.042 |
| 5   | 1.903 | 6.343 | 2.139 | 0.805 | 0.476 | 0.894 |
| 10  | 0.832 | 3.068 | 9.327 | 1.460 | 0.644 | 0.589 |
| 15  | 0.533 | 1.069 | 3.474 | 8.356 | 1.573 | 0.726 |
| 20  | 0.732 | 0.671 | 0.757 | 3.002 | 3.701 | 1.617 |
| 25  | 1.308 | 0.778 | 0.521 | 1.165 | 2.331 | 2.602 |

The matrices are square, with rows and columns labeled by the start age of each age group, so you can pick out values by age:

``` python
matrix.loc[25, 30]
```

    1.62107

|  | source | access | shape | bins | countries | study |
|----|----|----|----|----|----|----|
| 0 | prem | bundled | 16x16 | 5-year, 0-4 ... 75+ | 187 | prem2021, with prem2017 for the ten locations ... |
| 1 | prem2021 | bundled | 16x16 | 5-year, 0-4 ... 75+ | 177 | Prem et al. 2021 |
| 2 | prem2017 | fetched | 16x16 | 5-year, 0-4 ... 75+ | 152 | Prem et al. 2017 |
| 3 | mistry2021 | fetched | 85x85 | single-year, 0 ... 84+ | 36 | Mistry et al. 2021 |
| 4 | mistry2021_us | fetched | 85x85 | single-year, 0 ... 84+ | 51 US states (with DC) | Mistry et al. 2021 |

The `kind` argument selects the setting:

``` python
{kind: round(float(ssd.load('kenya', kind=kind).to_numpy().sum()), 1)
 for kind in ('all', 'home', 'work', 'school', 'community')}
```

    {'all': 218.1, 'home': 81.7, 'work': 38.2, 'school': 14.2, 'community': 84.0}

The `all` matrix is the combined one and the other four are its parts, although they don't add up exactly to `all`, since the original studies estimate them separately.

Coverage differs between sources, so it is worth checking. You can list the locations of a fetched source without downloading it:

``` python
len(ssd.list_locations('prem2021')), len(ssd.list_locations('prem2017')), len(ssd.list_locations('mistry2021'))
```

    (177, 152, 36)

The other two sources are read with [fetch()](../api/fetching.md#starsim_data.fetching.fetch), which downloads each one once (about 3 MB) and returns the same kind of frame as [load()](../api/loaders.md#starsim_data.loaders.load):

``` python
ssd.fetch('japan', which='prem2017', kind='home')   # 16x16
ssd.fetch('france', which='mistry2021')             # 85x85, single years of age
```

### Which source

`prem2021` is the authors' own update of `prem2017`, based on more contact surveys and newer demographic data, and covering 177 locations rather than 152. Where both have a country, they agree closely: the median correlation between a country's two matrices is 0.957.

`prem2021` dropped ten locations that `prem2017` has: Andorra, Antigua and Barbuda, Australia, Haiti, Japan, Kiribati, Lebanon, Monaco, Seychelles and Taiwan. Their `prem2017` matrices are bundled too, and the default contact matrices, `prem`, use them, so `ssd.load()` has a matrix for all 187 locations. Each matrix records which study it came from:

``` python
ssd.load('Japan').attrs['dataset']
```

    'prem2017'

Asking for `prem2021` by name gives that study alone, and the error says where else to find a location it lacks:

``` python
ssd.load('Japan', which='prem2021')
```

    ---------------------------------------------------------------------------
    KeyError                                  Traceback (most recent call last)
    Cell In[21], line 1
    ----> 1 ssd.load('Japan', which='prem2021')

    File ~/work/starsim_data/starsim_data/starsim_data/loaders.py:290, in load(location, which, kind, revision, level)
        288 if ds.tier > 0: # `load` never touches the network; say so rather than quietly downloading
        289     raise ValueError(f'{which!r} is not bundled; use {ds.function}({location!r}, which={which!r}) to download it')
    --> 290 return _read(which, location, kind=kind, revision=revision, level=level)

    File ~/work/starsim_data/starsim_data/starsim_data/loaders.py:311, in _read(which, location, kind, revision, level, path)
        309 with f:
        310     spec  = ssd.registry._spec(group)
    --> 311     key   = ssd.resolve(location, valid=group.keys(), dataset=which)
        312     opts  = dict(kind=kind, level=level)
        313     shape = ssd.registry.get_layout(spec['layout'])

    File ~/work/starsim_data/starsim_data/starsim_data/locations.py:203, in resolve(location, valid, dataset)
        201     hint  = f'; see list_locations({dataset!r})'
        202 errormsg = f'location {what} is not available{where} ({len(valid)} locations{hint}){_elsewhere(canonical, dataset)}'
    --> 203 raise KeyError(errormsg)

    KeyError: "location 'Japan' is not available in 'prem2021' (177 locations; see list_locations('prem2021')); it is in load('Japan') or fetch('Japan', which='mistry2021') or fetch('Japan', which='prem2017')"

The full `prem2017` is mainly useful for reproducing earlier analyses. `mistry2021` meets a different need: single-year age groups, but for only 36 mostly high-income countries, under the more permissive Apache-2.0 license.

### US states

`mistry2021_us` has the Mistry et al. 2021 matrices for each US state and the District of Columbia, and `population_us` has each one's population in 2023 by single year of age, from the US Census Bureau. A state is named with its country, separated by a colon, as in the node table's `dot_name` (see [Locations](locations.md#subnational-units)):

``` python
ssd.list_locations('mistry2021_us')[:3]
```

    ['United_States:Alabama', 'United_States:Alaska', 'United_States:Arizona']

``` python
ssd.fetch('United States:Massachusetts', which='mistry2021_us', kind='school') # 85x85
ssd.fetch('United States:Massachusetts', which='population_us') # Time, Sex, AgeGrp, Value
```

`population_us` has the same columns as `population`, for 2023 only and with `Sex` equal to `'Both sexes'`, since the source does not split by sex. Asking a national dataset for a state tells you where to find it:

``` python
ssd.load('United States:Massachusetts')
```

    ---------------------------------------------------------------------------
    KeyError                                  Traceback (most recent call last)
    Cell In[23], line 1
    ----> 1 ssd.load('United States:Massachusetts')

    File ~/work/starsim_data/starsim_data/starsim_data/loaders.py:285, in load(location, which, kind, revision, level)
        283 ds = ssd.registry.dataset(which)
        284 if ds.loader is not None:  # a registered dataset that is not a bundled store
    --> 285     return ds.loader(location, kind=kind, revision=revision, level=level)
        286 if (level is not None) and (ds.layout != 'nodes'):
        287     raise ValueError(f'level={level!r} does not apply to {which!r}; only the node table is addressed by administrative level')

    File ~/work/starsim_data/starsim_data/starsim_data/loaders.py:303, in _load_prem(location, kind, revision, level)
        301 if key in list_locations('prem'):
        302     return _read('prem2017', key, kind=kind, path=_data_file(ssd.registry.PREM['fill']))
    --> 303 return _read('prem2021', location, kind=kind)

    File ~/work/starsim_data/starsim_data/starsim_data/loaders.py:311, in _read(which, location, kind, revision, level, path)
        309 with f:
        310     spec  = ssd.registry._spec(group)
    --> 311     key   = ssd.resolve(location, valid=group.keys(), dataset=which)
        312     opts  = dict(kind=kind, level=level)
        313     shape = ssd.registry.get_layout(spec['layout'])

    File ~/work/starsim_data/starsim_data/starsim_data/locations.py:203, in resolve(location, valid, dataset)
        201     hint  = f'; see list_locations({dataset!r})'
        202 errormsg = f'location {what} is not available{where} ({len(valid)} locations{hint}){_elsewhere(canonical, dataset)}'
    --> 203 raise KeyError(errormsg)

    KeyError: "location 'United States:Massachusetts' is not available in 'prem2021' (177 locations; see list_locations('prem2021')); it is in fetch('United_States:Massachusetts', which='mistry2021_us')"

## The node table

The node table has one row per administrative unit (province, state, district and so on). It is described in full in [Spatial data](spatial.md); its columns are listed here too.

``` python
nodes = ssd.load_nodes('kenya', level=1)
nodes.head(3)
```

|  | nodeid | location | iso3 | level | name | pcode | dot_name | parent | lat | lon | area_km2 | population | source |
|----|----|----|----|----|----|----|----|----|----|----|----|----|----|
| 0 | 0 | Kenya | KEN | 1 | Baringo | KE030 | Kenya:Baringo | Kenya | 0.671607 | 35.945244 | 10889.239482 | 6.873870e+05 | unocha |
| 1 | 1 | Kenya | KEN | 1 | Bomet | KE036 | Kenya:Bomet | Kenya | -0.716578 | 35.298061 | 2438.219921 | 9.133519e+05 | unocha |
| 2 | 2 | Kenya | KEN | 1 | Bungoma | KE039 | Kenya:Bungoma | Kenya | 0.751052 | 34.640357 | 3005.749664 | 1.747200e+06 | unocha |

|     | column     | dtype   | note                                              |
|-----|------------|---------|---------------------------------------------------|
| 0   | nodeid     | int32   | numbered from 0 within each country and level;... |
| 1   | location   | str     | the country, e.g. Kenya                           |
| 2   | iso3       | str     | ISO 3166-1 alpha-3                                |
| 3   | level      | int32   | 0 = country, 1 = first-level subdivision          |
| 4   | name       | str     | the unit's own name, as the source spells it      |
| 5   | pcode      | str     | a standard code for the unit, for joining to o... |
| 6   | dot_name   | str     | hierarchical path, e.g. Kenya:Baringo             |
| 7   | parent     | str     | the parent's dot_name; empty at level 0           |
| 8   | lat        | float64 | latitude of the center of the unit                |
| 9   | lon        | float64 | longitude of the center of the unit               |
| 10  | area_km2   | float64 | area in square kilometers                         |
| 11  | population | float64 | UN WPP at level 0; WorldPop 2020 summed within... |
| 12  | source     | str     | where the boundaries came from: unocha or geob... |

## Beyond the bundle

For the contact sources that are not bundled (`prem2017`, `mistry2021` and `mistry2021_us`), [fetch()](../api/fetching.md#starsim_data.fetching.fetch) returns the same kind of frame [load()](../api/loaders.md#starsim_data.loaders.load) returns for `prem2021`. For the five bundled demographic series, [download()](../api/downloading.md#starsim_data.downloading.download) returns the same columns for every year, straight from the UN, and it also provides two datasets that are not bundled at all:

``` python
ssd.list_fetchable(), ssd.list_downloadable()
```

    (['mistry2021', 'mistry2021_us', 'population_us', 'prem2017'],
     ['births',
      'deaths',
      'fertility',
      'indicators',
      'life_table',
      'migration',
      'population'])

``` python
ssd.download('kenya', which='deaths', years=(2000, 2025))
```

| Returns | Columns |
|----|----|
| `births`, `migration` | as bundled, but every year |
| `deaths`, `population`, `fertility` | as bundled, but every year |
| `life_table` | `Time, Sex, AgeGrpStart, mx, qx, px, lx, dx, Lx, Sx, Tx, ex, ax`, for Female, Male and `Total` |
| `indicators` | `Time` plus ~50 WPP indicators: `CDR`, `TFR`, `LEx`, `IMR`, `Q5`, `NetMigrations`, … |

The `life_table` result comes with a glossary of its columns, since the standard two-letter names are easy to mix up:

``` python
lt = ssd.download('kenya', which='life_table', years=2020)
lt.attrs['columns']['Lx']
# 'person-years lived between ages x and x+n'
```

## What is not here

- **Subnational demographics,** other than the populations of the US states (`population_us`, above). WPP has no subnational dimension, so none can be derived from it.
- **Age-specific migration.** WPP publishes only the crude national rate.
- **"Both sexes" rows** in the bundled data, which has Female and Male only. `download(..., which='life_table')` does include `Total`.
- **Hong Kong, Macao and Puerto Rico** in the node table, since no boundary source treats them separately. Their demographics and contact matrices are included.
