Skip to main content

Datasets and their shapes

🧠15% 🤖85%

This page lists the columns and units of each dataset. The examples are run when this page is built, so they show what the current version actually returns.

import starsim_data as ssd

ssd.list_datasets(detailed='full').drop(columns='license')
dataset description layout access levels locations revisions
0 adjacency Admin units that share a border adjacency bundled 1 184
1 births Crude birth rate (UN WPP) crude bundled 0 187 2024
2 deaths Mortality by age and sex (UN WPP) mortality bundled 0 187 2024
3 fertility Fertility by age of mother (UN WPP) fertility bundled 0 187 2024
4 indicators All demographic indicators (UN WPP) wide downloaded 0 0
5 life_table Abridged life tables (UN WPP) long downloaded 0 0
6 migration Net migration rate (UN WPP) crude bundled 0 187 2024
7 mistry2021 Contact matrices (Mistry 2021) matrix fetched 0 36
8 mistry2021_us US contact matrices (Mistry 2021) matrix fetched 1 51
9 nodes Admin units: centroids, areas, codes nodes bundled 0, 1 184
10 population Population by age and sex (UN WPP) population bundled 0 187 2024
11 population_us US population by age (Census) population fetched 1 51
12 prem Contact matrices (Prem 2021 + 2017) matrix bundled 0 187
13 prem2017 Contact matrices (Prem 2017) matrix fetched 0 152
14 prem2021 Contact matrices (Prem 2021) matrix bundled 0 177

layout is how the data is shaped (the sections below describe each), levels the administrative levels it covers (0 for countries, 1 for states or provinces), and revisions the WPP revisions it is available for. Licenses are left out here to save space; they are listed in Provenance, and in full in NOTICE.md.

Demographics

Five series come from UN World Population Prospects (WPP) 2024, Medium variant, with one year per decade from 1950 to 2100 (sixteen years in all). All 187 countries have all five series.

The column names are the UN’s own, so the data can be passed to Starsim’s demographic modules without renaming anything.

Births and migration

Both are national crude rates, one value per year, not broken down by age. The UN publishes no age breakdown for migration; births by age of mother are the fertility series, below.

births = ssd.load('kenya', which='births')
births.head()
Time CBR
0 1950 49.337002
1 1960 55.183998
2 1970 52.540001
3 1980 48.210999
4 1990 43.930000
column dtype units note
0 Time int64 calendar year decadal, 1950-2100
1 CBR float32 births per 1,000 population per year

The migration series has the same shape, with CNMR (crude net-migration rate per 1,000) in place of CBR. It is signed, with negative values indicating net emigration.

ssd.load('kenya', which='migration').head(3)
Time CNMR
0 1950 1.090
1 1960 0.277
2 1970 -1.149

Deaths

Age- and sex-specific mortality, in the usual abridged age groups (0, 1–4, 5–9, …, 100+). Only Female and Male are included, not the UN’s “Both sexes” rows, to match Starsim.

deaths = ssd.load('kenya', which='deaths')
deaths.head()
Time Sex AgeGrpStart mx
0 1950 Female 0 0.160465
1 1950 Female 1 0.055043
2 1950 Female 5 0.012113
3 1950 Female 10 0.006858
4 1950 Female 15 0.006908
column dtype units note
0 Time int64 calendar year
1 Sex string Female | Male no "Both sexes"
2 AgeGrpStart int64 years start of the bin: 0, 1, 5, 10, ..., 100
3 mx float32 deaths per person-year the central death rate m(x)
NoteDeath rates are not probabilities

mx is the central death rate, deaths per person-year lived, not the probability of dying in the age group (qx). The two are similar where mortality is low and differ where it is high. The other life-table columns, including qx, are available through download(), as described below.

len(deaths), deaths.Time.nunique(), deaths.AgeGrpStart.nunique()
(704, 16, 22)

That is sixteen years times two sexes times twenty-two age groups.

Population

Population by 5-year age group and sex, in persons. (The UN publishes these in thousands; they are converted here.)

population = ssd.load('kenya', which='population')
population.head()
Time Sex AgeGrp Value
0 1950 Female 0 546394.0
1 1950 Female 5 418403.5
2 1950 Female 10 390101.0
3 1950 Female 15 338169.0
4 1950 Female 20 258914.0
column dtype units note
0 Time int64 calendar year
1 Sex string Female | Male
2 AgeGrp int64 years start of the bin: 0, 5, ..., 100
3 Value float32 persons absolute count, not thousands

The national total is a quick check on the units:

f"{population[population.Time == 2020].Value.sum() / 1e6:.1f}M"
'52.2M'

Fertility

Age-specific fertility rates, by 5-year age group of the mother (10–14, …, 50–54). There is no Sex column, since the rates are per woman.

fertility = ssd.load('kenya', which='fertility')
fertility.head()
Time AgeGrpStart ASFR
0 1950 10 23.204000
1 1950 15 162.516006
2 1950 20 280.062988
3 1950 25 302.570007
4 1950 30 293.190002
column dtype units note
0 Time int64 calendar year
1 AgeGrpStart int64 years start of the bin: 10, 15, ..., 50
2 ASFR float32 births per 1,000 women per year the age-specific fertility rate

Summed over the age groups, times five years per group and divided by 1,000, it gives the total fertility rate:

f"{5 * fertility[fertility.Time == 2020].ASFR.sum() / 1000:.2f}"
'3.37'

Revisions

The UN updates WPP every two years or so. Each release (revision) is kept separately, so a new one is added alongside the old rather than changing data already in use:

ssd.list_revisions()
['2024']
ssd.load('kenya', which='deaths')                 # newest bundled revision
ssd.load('kenya', which='deaths', revision=2024)  # pinned

Naming the revision is a good idea in any analysis you want to be able to reproduce later.

Contact matrices

Contact matrices give the average number of daily contacts between people in each pair of age groups. Three sources are available, each with five contact settings, selected with the kind argument. One of them, prem2021, is bundled; the other two are fetched the first time you use them (see Bundled, fetched and downloaded data). The default, prem, is prem2021, or prem2017 for the ten locations prem2021 lacks (see Which source).

matrix = ssd.load('kenya')   # prem2021
matrix.iloc[:6, :6].round(3)
0 5 10 15 20 25
0 2.402 1.808 1.109 0.656 0.812 1.042
5 1.903 6.343 2.139 0.805 0.476 0.894
10 0.832 3.068 9.327 1.460 0.644 0.589
15 0.533 1.069 3.474 8.356 1.573 0.726
20 0.732 0.671 0.757 3.002 3.701 1.617
25 1.308 0.778 0.521 1.165 2.331 2.602

The matrices are square, with rows and columns labeled by the start age of each age group, so you can pick out values by age:

matrix.loc[25, 30]
1.62107
source access shape bins countries study
0 prem bundled 16x16 5-year, 0-4 ... 75+ 187 prem2021, with prem2017 for the ten locations ...
1 prem2021 bundled 16x16 5-year, 0-4 ... 75+ 177 Prem et al. 2021
2 prem2017 fetched 16x16 5-year, 0-4 ... 75+ 152 Prem et al. 2017
3 mistry2021 fetched 85x85 single-year, 0 ... 84+ 36 Mistry et al. 2021
4 mistry2021_us fetched 85x85 single-year, 0 ... 84+ 51 US states (with DC) Mistry et al. 2021

The kind argument selects the setting:

{kind: round(float(ssd.load('kenya', kind=kind).to_numpy().sum()), 1)
 for kind in ('all', 'home', 'work', 'school', 'community')}
{'all': 218.1, 'home': 81.7, 'work': 38.2, 'school': 14.2, 'community': 84.0}

The all matrix is the combined one and the other four are its parts, although they don’t add up exactly to all, since the original studies estimate them separately.

Coverage differs between sources, so it is worth checking. You can list the locations of a fetched source without downloading it:

len(ssd.list_locations('prem2021')), len(ssd.list_locations('prem2017')), len(ssd.list_locations('mistry2021'))
(177, 152, 36)

The other two sources are read with fetch(), which downloads each one once (about 3 MB) and returns the same kind of frame as load():

ssd.fetch('japan', which='prem2017', kind='home')   # 16x16
ssd.fetch('france', which='mistry2021')             # 85x85, single years of age

Which source

prem2021 is the authors’ own update of prem2017, based on more contact surveys and newer demographic data, and covering 177 locations rather than 152. Where both have a country, they agree closely: the median correlation between a country’s two matrices is 0.957.

prem2021 dropped ten locations that prem2017 has: Andorra, Antigua and Barbuda, Australia, Haiti, Japan, Kiribati, Lebanon, Monaco, Seychelles and Taiwan. Their prem2017 matrices are bundled too, and the default contact matrices, prem, use them, so ssd.load() has a matrix for all 187 locations. Each matrix records which study it came from:

ssd.load('Japan').attrs['dataset']
'prem2017'

Asking for prem2021 by name gives that study alone, and the error says where else to find a location it lacks:

ssd.load('Japan', which='prem2021')
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[21], line 1
----> 1 ssd.load('Japan', which='prem2021')

File ~/work/starsim_data/starsim_data/starsim_data/loaders.py:290, in load(location, which, kind, revision, level)
    288 if ds.tier > 0: # `load` never touches the network; say so rather than quietly downloading
    289     raise ValueError(f'{which!r} is not bundled; use {ds.function}({location!r}, which={which!r}) to download it')
--> 290 return _read(which, location, kind=kind, revision=revision, level=level)

File ~/work/starsim_data/starsim_data/starsim_data/loaders.py:311, in _read(which, location, kind, revision, level, path)
    309 with f:
    310     spec  = ssd.registry._spec(group)
--> 311     key   = ssd.resolve(location, valid=group.keys(), dataset=which)
    312     opts  = dict(kind=kind, level=level)
    313     shape = ssd.registry.get_layout(spec['layout'])

File ~/work/starsim_data/starsim_data/starsim_data/locations.py:203, in resolve(location, valid, dataset)
    201     hint  = f'; see list_locations({dataset!r})'
    202 errormsg = f'location {what} is not available{where} ({len(valid)} locations{hint}){_elsewhere(canonical, dataset)}'
--> 203 raise KeyError(errormsg)

KeyError: "location 'Japan' is not available in 'prem2021' (177 locations; see list_locations('prem2021')); it is in load('Japan') or fetch('Japan', which='mistry2021') or fetch('Japan', which='prem2017')"

The full prem2017 is mainly useful for reproducing earlier analyses. mistry2021 meets a different need: single-year age groups, but for only 36 mostly high-income countries, under the more permissive Apache-2.0 license.

US states

mistry2021_us has the Mistry et al. 2021 matrices for each US state and the District of Columbia, and population_us has each one’s population in 2023 by single year of age, from the US Census Bureau. A state is named with its country, separated by a colon, as in the node table’s dot_name (see Locations):

ssd.list_locations('mistry2021_us')[:3]
['United_States:Alabama', 'United_States:Alaska', 'United_States:Arizona']
ssd.fetch('United States:Massachusetts', which='mistry2021_us', kind='school') # 85x85
ssd.fetch('United States:Massachusetts', which='population_us') # Time, Sex, AgeGrp, Value

population_us has the same columns as population, for 2023 only and with Sex equal to 'Both sexes', since the source does not split by sex. Asking a national dataset for a state tells you where to find it:

ssd.load('United States:Massachusetts')
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[23], line 1
----> 1 ssd.load('United States:Massachusetts')

File ~/work/starsim_data/starsim_data/starsim_data/loaders.py:285, in load(location, which, kind, revision, level)
    283 ds = ssd.registry.dataset(which)
    284 if ds.loader is not None:  # a registered dataset that is not a bundled store
--> 285     return ds.loader(location, kind=kind, revision=revision, level=level)
    286 if (level is not None) and (ds.layout != 'nodes'):
    287     raise ValueError(f'level={level!r} does not apply to {which!r}; only the node table is addressed by administrative level')

File ~/work/starsim_data/starsim_data/starsim_data/loaders.py:303, in _load_prem(location, kind, revision, level)
    301 if key in list_locations('prem'):
    302     return _read('prem2017', key, kind=kind, path=_data_file(ssd.registry.PREM['fill']))
--> 303 return _read('prem2021', location, kind=kind)

File ~/work/starsim_data/starsim_data/starsim_data/loaders.py:311, in _read(which, location, kind, revision, level, path)
    309 with f:
    310     spec  = ssd.registry._spec(group)
--> 311     key   = ssd.resolve(location, valid=group.keys(), dataset=which)
    312     opts  = dict(kind=kind, level=level)
    313     shape = ssd.registry.get_layout(spec['layout'])

File ~/work/starsim_data/starsim_data/starsim_data/locations.py:203, in resolve(location, valid, dataset)
    201     hint  = f'; see list_locations({dataset!r})'
    202 errormsg = f'location {what} is not available{where} ({len(valid)} locations{hint}){_elsewhere(canonical, dataset)}'
--> 203 raise KeyError(errormsg)

KeyError: "location 'United States:Massachusetts' is not available in 'prem2021' (177 locations; see list_locations('prem2021')); it is in fetch('United_States:Massachusetts', which='mistry2021_us')"

The node table

The node table has one row per administrative unit (province, state, district and so on). It is described in full in Spatial data; its columns are listed here too.

nodes = ssd.load_nodes('kenya', level=1)
nodes.head(3)
nodeid location iso3 level name pcode dot_name parent lat lon area_km2 population source
0 0 Kenya KEN 1 Baringo KE030 Kenya:Baringo Kenya 0.671607 35.945244 10889.239482 6.873870e+05 unocha
1 1 Kenya KEN 1 Bomet KE036 Kenya:Bomet Kenya -0.716578 35.298061 2438.219921 9.133519e+05 unocha
2 2 Kenya KEN 1 Bungoma KE039 Kenya:Bungoma Kenya 0.751052 34.640357 3005.749664 1.747200e+06 unocha
column dtype note
0 nodeid int32 numbered from 0 within each country and level;...
1 location str the country, e.g. Kenya
2 iso3 str ISO 3166-1 alpha-3
3 level int32 0 = country, 1 = first-level subdivision
4 name str the unit's own name, as the source spells it
5 pcode str a standard code for the unit, for joining to o...
6 dot_name str hierarchical path, e.g. Kenya:Baringo
7 parent str the parent's dot_name; empty at level 0
8 lat float64 latitude of the center of the unit
9 lon float64 longitude of the center of the unit
10 area_km2 float64 area in square kilometers
11 population float64 UN WPP at level 0; WorldPop 2020 summed within...
12 source str where the boundaries came from: unocha or geob...

Beyond the bundle

For the contact sources that are not bundled (prem2017, mistry2021 and mistry2021_us), fetch() returns the same kind of frame load() returns for prem2021. For the five bundled demographic series, download() returns the same columns for every year, straight from the UN, and it also provides two datasets that are not bundled at all:

ssd.list_fetchable(), ssd.list_downloadable()
(['mistry2021', 'mistry2021_us', 'population_us', 'prem2017'],
 ['births',
  'deaths',
  'fertility',
  'indicators',
  'life_table',
  'migration',
  'population'])
ssd.download('kenya', which='deaths', years=(2000, 2025))
Returns Columns
births, migration as bundled, but every year
deaths, population, fertility as bundled, but every year
life_table Time, Sex, AgeGrpStart, mx, qx, px, lx, dx, Lx, Sx, Tx, ex, ax, for Female, Male and Total
indicators Time plus ~50 WPP indicators: CDR, TFR, LEx, IMR, Q5, NetMigrations, …

The life_table result comes with a glossary of its columns, since the standard two-letter names are easy to mix up:

lt = ssd.download('kenya', which='life_table', years=2020)
lt.attrs['columns']['Lx']
# 'person-years lived between ages x and x+n'

What is not here

  • Subnational demographics, other than the populations of the US states (population_us, above). WPP has no subnational dimension, so none can be derived from it.
  • Age-specific migration. WPP publishes only the crude national rate.
  • “Both sexes” rows in the bundled data, which has Female and Male only. download(..., which='life_table') does include Total.
  • Hong Kong, Macao and Puerto Rico in the node table, since no boundary source treats them separately. Their demographics and contact matrices are included.