🧠15% 🤖85%
This page lists the columns and units of each dataset. The examples are run when this page is built, so they show what the current version actually returns.
import starsim_data as ssd
ssd.list_datasets(detailed='full').drop(columns='license')
|
dataset |
description |
layout |
access |
levels |
locations |
revisions |
| 0 |
adjacency |
Admin units that share a border |
adjacency |
bundled |
1 |
184 |
|
| 1 |
births |
Crude birth rate (UN WPP) |
crude |
bundled |
0 |
187 |
2024 |
| 2 |
deaths |
Mortality by age and sex (UN WPP) |
mortality |
bundled |
0 |
187 |
2024 |
| 3 |
fertility |
Fertility by age of mother (UN WPP) |
fertility |
bundled |
0 |
187 |
2024 |
| 4 |
indicators |
All demographic indicators (UN WPP) |
wide |
downloaded |
0 |
0 |
|
| 5 |
life_table |
Abridged life tables (UN WPP) |
long |
downloaded |
0 |
0 |
|
| 6 |
migration |
Net migration rate (UN WPP) |
crude |
bundled |
0 |
187 |
2024 |
| 7 |
mistry2021 |
Contact matrices (Mistry 2021) |
matrix |
fetched |
0 |
36 |
|
| 8 |
mistry2021_us |
US contact matrices (Mistry 2021) |
matrix |
fetched |
1 |
51 |
|
| 9 |
nodes |
Admin units: centroids, areas, codes |
nodes |
bundled |
0, 1 |
184 |
|
| 10 |
population |
Population by age and sex (UN WPP) |
population |
bundled |
0 |
187 |
2024 |
| 11 |
population_us |
US population by age (Census) |
population |
fetched |
1 |
51 |
|
| 12 |
prem |
Contact matrices (Prem 2021 + 2017) |
matrix |
bundled |
0 |
187 |
|
| 13 |
prem2017 |
Contact matrices (Prem 2017) |
matrix |
fetched |
0 |
152 |
|
| 14 |
prem2021 |
Contact matrices (Prem 2021) |
matrix |
bundled |
0 |
177 |
|
layout is how the data is shaped (the sections below describe each), levels the administrative levels it covers (0 for countries, 1 for states or provinces), and revisions the WPP revisions it is available for. Licenses are left out here to save space; they are listed in Provenance, and in full in NOTICE.md.
Demographics
Five series come from UN World Population Prospects (WPP) 2024, Medium variant, with one year per decade from 1950 to 2100 (sixteen years in all). All 187 countries have all five series.
The column names are the UN’s own, so the data can be passed to Starsim’s demographic modules without renaming anything.
Births and migration
Both are national crude rates, one value per year, not broken down by age. The UN publishes no age breakdown for migration; births by age of mother are the fertility series, below.
births = ssd.load('kenya', which='births')
births.head()
|
Time |
CBR |
| 0 |
1950 |
49.337002 |
| 1 |
1960 |
55.183998 |
| 2 |
1970 |
52.540001 |
| 3 |
1980 |
48.210999 |
| 4 |
1990 |
43.930000 |
|
column |
dtype |
units |
note |
| 0 |
Time |
int64 |
calendar year |
decadal, 1950-2100 |
| 1 |
CBR |
float32 |
births per 1,000 population per year |
|
The migration series has the same shape, with CNMR (crude net-migration rate per 1,000) in place of CBR. It is signed, with negative values indicating net emigration.
ssd.load('kenya', which='migration').head(3)
|
Time |
CNMR |
| 0 |
1950 |
1.090 |
| 1 |
1960 |
0.277 |
| 2 |
1970 |
-1.149 |
Deaths
Age- and sex-specific mortality, in the usual abridged age groups (0, 1–4, 5–9, …, 100+). Only Female and Male are included, not the UN’s “Both sexes” rows, to match Starsim.
deaths = ssd.load('kenya', which='deaths')
deaths.head()
|
Time |
Sex |
AgeGrpStart |
mx |
| 0 |
1950 |
Female |
0 |
0.160465 |
| 1 |
1950 |
Female |
1 |
0.055043 |
| 2 |
1950 |
Female |
5 |
0.012113 |
| 3 |
1950 |
Female |
10 |
0.006858 |
| 4 |
1950 |
Female |
15 |
0.006908 |
|
column |
dtype |
units |
note |
| 0 |
Time |
int64 |
calendar year |
|
| 1 |
Sex |
string |
Female | Male |
no "Both sexes" |
| 2 |
AgeGrpStart |
int64 |
years |
start of the bin: 0, 1, 5, 10, ..., 100 |
| 3 |
mx |
float32 |
deaths per person-year |
the central death rate m(x) |
mx is the central death rate, deaths per person-year lived, not the probability of dying in the age group (qx). The two are similar where mortality is low and differ where it is high. The other life-table columns, including qx, are available through download(), as described below.
len(deaths), deaths.Time.nunique(), deaths.AgeGrpStart.nunique()
That is sixteen years times two sexes times twenty-two age groups.
Population
Population by 5-year age group and sex, in persons. (The UN publishes these in thousands; they are converted here.)
population = ssd.load('kenya', which='population')
population.head()
|
Time |
Sex |
AgeGrp |
Value |
| 0 |
1950 |
Female |
0 |
546394.0 |
| 1 |
1950 |
Female |
5 |
418403.5 |
| 2 |
1950 |
Female |
10 |
390101.0 |
| 3 |
1950 |
Female |
15 |
338169.0 |
| 4 |
1950 |
Female |
20 |
258914.0 |
|
column |
dtype |
units |
note |
| 0 |
Time |
int64 |
calendar year |
|
| 1 |
Sex |
string |
Female | Male |
|
| 2 |
AgeGrp |
int64 |
years |
start of the bin: 0, 5, ..., 100 |
| 3 |
Value |
float32 |
persons |
absolute count, not thousands |
The national total is a quick check on the units:
f"{population[population.Time == 2020].Value.sum() / 1e6:.1f}M"
Fertility
Age-specific fertility rates, by 5-year age group of the mother (10–14, …, 50–54). There is no Sex column, since the rates are per woman.
fertility = ssd.load('kenya', which='fertility')
fertility.head()
|
Time |
AgeGrpStart |
ASFR |
| 0 |
1950 |
10 |
23.204000 |
| 1 |
1950 |
15 |
162.516006 |
| 2 |
1950 |
20 |
280.062988 |
| 3 |
1950 |
25 |
302.570007 |
| 4 |
1950 |
30 |
293.190002 |
|
column |
dtype |
units |
note |
| 0 |
Time |
int64 |
calendar year |
|
| 1 |
AgeGrpStart |
int64 |
years |
start of the bin: 10, 15, ..., 50 |
| 2 |
ASFR |
float32 |
births per 1,000 women per year |
the age-specific fertility rate |
Summed over the age groups, times five years per group and divided by 1,000, it gives the total fertility rate:
f"{5 * fertility[fertility.Time == 2020].ASFR.sum() / 1000:.2f}"
Revisions
The UN updates WPP every two years or so. Each release (revision) is kept separately, so a new one is added alongside the old rather than changing data already in use:
ssd.load('kenya', which='deaths') # newest bundled revision
ssd.load('kenya', which='deaths', revision=2024) # pinned
Naming the revision is a good idea in any analysis you want to be able to reproduce later.
Contact matrices
Contact matrices give the average number of daily contacts between people in each pair of age groups. Three sources are available, each with five contact settings, selected with the kind argument. One of them, prem2021, is bundled; the other two are fetched the first time you use them (see Bundled, fetched and downloaded data). The default, prem, is prem2021, or prem2017 for the ten locations prem2021 lacks (see Which source).
matrix = ssd.load('kenya') # prem2021
matrix.iloc[:6, :6].round(3)
|
0 |
5 |
10 |
15 |
20 |
25 |
| 0 |
2.402 |
1.808 |
1.109 |
0.656 |
0.812 |
1.042 |
| 5 |
1.903 |
6.343 |
2.139 |
0.805 |
0.476 |
0.894 |
| 10 |
0.832 |
3.068 |
9.327 |
1.460 |
0.644 |
0.589 |
| 15 |
0.533 |
1.069 |
3.474 |
8.356 |
1.573 |
0.726 |
| 20 |
0.732 |
0.671 |
0.757 |
3.002 |
3.701 |
1.617 |
| 25 |
1.308 |
0.778 |
0.521 |
1.165 |
2.331 |
2.602 |
The matrices are square, with rows and columns labeled by the start age of each age group, so you can pick out values by age:
|
source |
access |
shape |
bins |
countries |
study |
| 0 |
prem |
bundled |
16x16 |
5-year, 0-4 ... 75+ |
187 |
prem2021, with prem2017 for the ten locations ... |
| 1 |
prem2021 |
bundled |
16x16 |
5-year, 0-4 ... 75+ |
177 |
Prem et al. 2021 |
| 2 |
prem2017 |
fetched |
16x16 |
5-year, 0-4 ... 75+ |
152 |
Prem et al. 2017 |
| 3 |
mistry2021 |
fetched |
85x85 |
single-year, 0 ... 84+ |
36 |
Mistry et al. 2021 |
| 4 |
mistry2021_us |
fetched |
85x85 |
single-year, 0 ... 84+ |
51 US states (with DC) |
Mistry et al. 2021 |
The kind argument selects the setting:
{kind: round(float(ssd.load('kenya', kind=kind).to_numpy().sum()), 1)
for kind in ('all', 'home', 'work', 'school', 'community')}
{'all': 218.1, 'home': 81.7, 'work': 38.2, 'school': 14.2, 'community': 84.0}
The all matrix is the combined one and the other four are its parts, although they don’t add up exactly to all, since the original studies estimate them separately.
Coverage differs between sources, so it is worth checking. You can list the locations of a fetched source without downloading it:
len(ssd.list_locations('prem2021')), len(ssd.list_locations('prem2017')), len(ssd.list_locations('mistry2021'))
The other two sources are read with fetch(), which downloads each one once (about 3 MB) and returns the same kind of frame as load():
ssd.fetch('japan', which='prem2017', kind='home') # 16x16
ssd.fetch('france', which='mistry2021') # 85x85, single years of age
Which source
prem2021 is the authors’ own update of prem2017, based on more contact surveys and newer demographic data, and covering 177 locations rather than 152. Where both have a country, they agree closely: the median correlation between a country’s two matrices is 0.957.
prem2021 dropped ten locations that prem2017 has: Andorra, Antigua and Barbuda, Australia, Haiti, Japan, Kiribati, Lebanon, Monaco, Seychelles and Taiwan. Their prem2017 matrices are bundled too, and the default contact matrices, prem, use them, so ssd.load() has a matrix for all 187 locations. Each matrix records which study it came from:
ssd.load('Japan').attrs['dataset']
Asking for prem2021 by name gives that study alone, and the error says where else to find a location it lacks:
ssd.load('Japan', which='prem2021')
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
Cell In[21], line 1
----> 1 ssd.load('Japan', which='prem2021')
File ~/work/starsim_data/starsim_data/starsim_data/loaders.py:290, in load(location, which, kind, revision, level)
288 if ds.tier > 0: # `load` never touches the network; say so rather than quietly downloading
289 raise ValueError(f'{which!r} is not bundled; use {ds.function}({location!r}, which={which!r}) to download it')
--> 290 return _read(which, location, kind=kind, revision=revision, level=level)
File ~/work/starsim_data/starsim_data/starsim_data/loaders.py:311, in _read(which, location, kind, revision, level, path)
309 with f:
310 spec = ssd.registry._spec(group)
--> 311 key = ssd.resolve(location, valid=group.keys(), dataset=which)
312 opts = dict(kind=kind, level=level)
313 shape = ssd.registry.get_layout(spec['layout'])
File ~/work/starsim_data/starsim_data/starsim_data/locations.py:203, in resolve(location, valid, dataset)
201 hint = f'; see list_locations({dataset!r})'
202 errormsg = f'location {what} is not available{where} ({len(valid)} locations{hint}){_elsewhere(canonical, dataset)}'
--> 203 raise KeyError(errormsg)
KeyError: "location 'Japan' is not available in 'prem2021' (177 locations; see list_locations('prem2021')); it is in load('Japan') or fetch('Japan', which='mistry2021') or fetch('Japan', which='prem2017')"
The full prem2017 is mainly useful for reproducing earlier analyses. mistry2021 meets a different need: single-year age groups, but for only 36 mostly high-income countries, under the more permissive Apache-2.0 license.
US states
mistry2021_us has the Mistry et al. 2021 matrices for each US state and the District of Columbia, and population_us has each one’s population in 2023 by single year of age, from the US Census Bureau. A state is named with its country, separated by a colon, as in the node table’s dot_name (see Locations):
ssd.list_locations('mistry2021_us')[:3]
['United_States:Alabama', 'United_States:Alaska', 'United_States:Arizona']
ssd.fetch('United States:Massachusetts', which='mistry2021_us', kind='school') # 85x85
ssd.fetch('United States:Massachusetts', which='population_us') # Time, Sex, AgeGrp, Value
population_us has the same columns as population, for 2023 only and with Sex equal to 'Both sexes', since the source does not split by sex. Asking a national dataset for a state tells you where to find it:
ssd.load('United States:Massachusetts')
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
Cell In[23], line 1
----> 1 ssd.load('United States:Massachusetts')
File ~/work/starsim_data/starsim_data/starsim_data/loaders.py:285, in load(location, which, kind, revision, level)
283 ds = ssd.registry.dataset(which)
284 if ds.loader is not None: # a registered dataset that is not a bundled store
--> 285 return ds.loader(location, kind=kind, revision=revision, level=level)
286 if (level is not None) and (ds.layout != 'nodes'):
287 raise ValueError(f'level={level!r} does not apply to {which!r}; only the node table is addressed by administrative level')
File ~/work/starsim_data/starsim_data/starsim_data/loaders.py:303, in _load_prem(location, kind, revision, level)
301 if key in list_locations('prem'):
302 return _read('prem2017', key, kind=kind, path=_data_file(ssd.registry.PREM['fill']))
--> 303 return _read('prem2021', location, kind=kind)
File ~/work/starsim_data/starsim_data/starsim_data/loaders.py:311, in _read(which, location, kind, revision, level, path)
309 with f:
310 spec = ssd.registry._spec(group)
--> 311 key = ssd.resolve(location, valid=group.keys(), dataset=which)
312 opts = dict(kind=kind, level=level)
313 shape = ssd.registry.get_layout(spec['layout'])
File ~/work/starsim_data/starsim_data/starsim_data/locations.py:203, in resolve(location, valid, dataset)
201 hint = f'; see list_locations({dataset!r})'
202 errormsg = f'location {what} is not available{where} ({len(valid)} locations{hint}){_elsewhere(canonical, dataset)}'
--> 203 raise KeyError(errormsg)
KeyError: "location 'United States:Massachusetts' is not available in 'prem2021' (177 locations; see list_locations('prem2021')); it is in fetch('United_States:Massachusetts', which='mistry2021_us')"
The node table
The node table has one row per administrative unit (province, state, district and so on). It is described in full in Spatial data; its columns are listed here too.
nodes = ssd.load_nodes('kenya', level=1)
nodes.head(3)
|
nodeid |
location |
iso3 |
level |
name |
pcode |
dot_name |
parent |
lat |
lon |
area_km2 |
population |
source |
| 0 |
0 |
Kenya |
KEN |
1 |
Baringo |
KE030 |
Kenya:Baringo |
Kenya |
0.671607 |
35.945244 |
10889.239482 |
6.873870e+05 |
unocha |
| 1 |
1 |
Kenya |
KEN |
1 |
Bomet |
KE036 |
Kenya:Bomet |
Kenya |
-0.716578 |
35.298061 |
2438.219921 |
9.133519e+05 |
unocha |
| 2 |
2 |
Kenya |
KEN |
1 |
Bungoma |
KE039 |
Kenya:Bungoma |
Kenya |
0.751052 |
34.640357 |
3005.749664 |
1.747200e+06 |
unocha |
|
column |
dtype |
note |
| 0 |
nodeid |
int32 |
numbered from 0 within each country and level;... |
| 1 |
location |
str |
the country, e.g. Kenya |
| 2 |
iso3 |
str |
ISO 3166-1 alpha-3 |
| 3 |
level |
int32 |
0 = country, 1 = first-level subdivision |
| 4 |
name |
str |
the unit's own name, as the source spells it |
| 5 |
pcode |
str |
a standard code for the unit, for joining to o... |
| 6 |
dot_name |
str |
hierarchical path, e.g. Kenya:Baringo |
| 7 |
parent |
str |
the parent's dot_name; empty at level 0 |
| 8 |
lat |
float64 |
latitude of the center of the unit |
| 9 |
lon |
float64 |
longitude of the center of the unit |
| 10 |
area_km2 |
float64 |
area in square kilometers |
| 11 |
population |
float64 |
UN WPP at level 0; WorldPop 2020 summed within... |
| 12 |
source |
str |
where the boundaries came from: unocha or geob... |
Beyond the bundle
For the contact sources that are not bundled (prem2017, mistry2021 and mistry2021_us), fetch() returns the same kind of frame load() returns for prem2021. For the five bundled demographic series, download() returns the same columns for every year, straight from the UN, and it also provides two datasets that are not bundled at all:
ssd.list_fetchable(), ssd.list_downloadable()
(['mistry2021', 'mistry2021_us', 'population_us', 'prem2017'],
['births',
'deaths',
'fertility',
'indicators',
'life_table',
'migration',
'population'])
ssd.download('kenya', which='deaths', years=(2000, 2025))
births, migration |
as bundled, but every year |
deaths, population, fertility |
as bundled, but every year |
life_table |
Time, Sex, AgeGrpStart, mx, qx, px, lx, dx, Lx, Sx, Tx, ex, ax, for Female, Male and Total |
indicators |
Time plus ~50 WPP indicators: CDR, TFR, LEx, IMR, Q5, NetMigrations, … |
The life_table result comes with a glossary of its columns, since the standard two-letter names are easy to mix up:
lt = ssd.download('kenya', which='life_table', years=2020)
lt.attrs['columns']['Lx']
# 'person-years lived between ages x and x+n'
What is not here
- Subnational demographics, other than the populations of the US states (
population_us, above). WPP has no subnational dimension, so none can be derived from it.
- Age-specific migration. WPP publishes only the crude national rate.
- “Both sexes” rows in the bundled data, which has Female and Male only.
download(..., which='life_table') does include Total.
- Hong Kong, Macao and Puerto Rico in the node table, since no boundary source treats them separately. Their demographics and contact matrices are included.