Skip to main content

Bundled, fetched and downloaded data

🧠15% 🤖85%

Every dataset in starsim_data is one of three kinds, and each kind has its own function, so you can tell from the call where the data comes from:

Comes from How you get it Internet needed Same for everyone on a version?
Bundled Inside the package load(), load_nodes(), distances(), adjacency() Never Yes
Fetched This package’s releases fetch() First use only Yes
Downloaded The original publisher download(), download_shapes(), download_nodes(), download_raster() First use only Unless the publisher changes the file

Fetched and downloaded data are both saved on your computer after the first use, so they need the internet only once. The difference is where they come from: fetch() gets files from this package’s own releases, while download() goes to the UN, WorldPop and the boundary publishers. Their servers are outside this package’s control, and may be slow, change their files or be down. Calling fetch() for data that isn’t packaged still works, but warns that it is downloading from an outside source.

The access column of list_datasets() says which kind each dataset is:

import starsim_data as ssd

ssd.list_datasets(detailed=True)
dataset description access locations
0 adjacency Admin units that share a border bundled 184
1 births Crude birth rate (UN WPP) bundled 187
2 deaths Mortality by age and sex (UN WPP) bundled 187
3 fertility Fertility by age of mother (UN WPP) bundled 187
4 indicators All demographic indicators (UN WPP) downloaded 0
5 life_table Abridged life tables (UN WPP) downloaded 0
6 migration Net migration rate (UN WPP) bundled 187
7 mistry2021 Contact matrices (Mistry 2021) fetched 36
8 mistry2021_us US contact matrices (Mistry 2021) fetched 51
9 nodes Admin units: centroids, areas, codes bundled 184
10 population Population by age and sex (UN WPP) bundled 187
11 population_us US population by age (Census) fetched 51
12 prem Contact matrices (Prem 2021 + 2017) bundled 187
13 prem2017 Contact matrices (Prem 2017) fetched 152
14 prem2021 Contact matrices (Prem 2021) bundled 177

Bundled

The demographic data (one year per decade, Medium variant), the default contact matrices (prem2021, plus prem2017 for the ten locations it lacks), the location table and the subnational node table are inside the package, about 9 MB in all. Everyone who installs a given version has exactly the same data, it works offline, and citing the version is enough to say which data an analysis used.

load() only ever reads bundled data. If you ask it for anything else, such as ssd.load('kenya', which='life_table'), it doesn’t download it; it gives an error telling you the fetch() or download() call to use instead.

Fetched

Some data this package prepares is too large to include, or needed by too few people:

  • The full prem2017 and mistry2021 contact matrices, about 3 MB each.
  • The US states’ contact matrices (mistry2021_us, 5 MB) and their populations by single year of age (population_us).

These are prepared the same way as the bundled data and published with each release of the package. fetch() downloads one the first time you use it, checks that it is the file that was published, and saves it. As with bundled data, everyone on a given version gets the same data, and you can list which locations a fetched dataset covers without downloading it.

Downloaded

Everything else comes straight from the organization that publishes it:

  • Annual UN WPP series, complete life tables, other projection variants and summary indicators, from the UN’s bulk files (17–290 MB each).
  • Administrative boundaries with geometry, from UNOCHA, geoBoundaries or GADM (tens of megabytes per country at admin level 2).
  • Gridded population rasters from WorldPop (1–40 MB per country-year), and the subnational populations download_nodes() computes from them.

These are also saved after the first use, along with a note of where and when each was downloaded. Since the files belong to their publishers, a publisher that updates one changes what a fresh download gives you; Provenance and reproducibility covers how to keep an analysis reproducible anyway.

The cache

Fetched and downloaded files are saved in one folder, the cache:

ssd.cache_dir()
PosixPath('/home/runner/.cache/starsim_data')
ssd.cache_info()      # what is cached, its source, and when it was retrieved
ssd.clear_cache()     # or clear_cache('wpp/*')

To put it somewhere else, set ssd.options.cache_dir or the STARSIM_DATA_CACHE_DIR environment variable.

Which kind something is

The split is by size and how widely the data is needed, not by subject. The same UN WPP 2024 release is both bundled and downloaded: five series at one year per decade take 4 MB and are bundled, while the complete annual life tables take 290 MB and are downloaded. Likewise, the contact matrices most models should use are bundled (see Datasets), and the others are fetched.

requirement call access cost
0 Age structure, mortality, births, fertility, c... load() bundled instant
1 Positions, areas and separations of administra... load_nodes(), distances() bundled instant
2 The other contact matrices (prem2017, mistry2021) fetch() fetched 3 MB each, once
3 US states: contact matrices and populations fetch() fetched 5 MB, once
4 Annual rates, life tables, other variants download() downloaded 17-290 MB, once
5 Polygon geometry, for mapping or spatial joins download_shapes() downloaded 1-50 MB per country
6 Subnational population estimates download_nodes() downloaded 1-40 MB per country

An analysis that only uses the first two rows needs no internet connection at all, and the package version is enough to say exactly which data it used.