🧠15% 🤖85%
Every dataset in starsim_data is one of three kinds, and each kind has its own function, so you can tell from the call where the data comes from:
Fetched and downloaded data are both saved on your computer after the first use, so they need the internet only once. The difference is where they come from: fetch() gets files from this package’s own releases, while download() goes to the UN, WorldPop and the boundary publishers. Their servers are outside this package’s control, and may be slow, change their files or be down. Calling fetch() for data that isn’t packaged still works, but warns that it is downloading from an outside source.
The access column of list_datasets() says which kind each dataset is:
import starsim_data as ssd
ssd.list_datasets(detailed=True)
|
dataset |
description |
access |
locations |
| 0 |
adjacency |
Admin units that share a border |
bundled |
184 |
| 1 |
births |
Crude birth rate (UN WPP) |
bundled |
187 |
| 2 |
deaths |
Mortality by age and sex (UN WPP) |
bundled |
187 |
| 3 |
fertility |
Fertility by age of mother (UN WPP) |
bundled |
187 |
| 4 |
indicators |
All demographic indicators (UN WPP) |
downloaded |
0 |
| 5 |
life_table |
Abridged life tables (UN WPP) |
downloaded |
0 |
| 6 |
migration |
Net migration rate (UN WPP) |
bundled |
187 |
| 7 |
mistry2021 |
Contact matrices (Mistry 2021) |
fetched |
36 |
| 8 |
mistry2021_us |
US contact matrices (Mistry 2021) |
fetched |
51 |
| 9 |
nodes |
Admin units: centroids, areas, codes |
bundled |
184 |
| 10 |
population |
Population by age and sex (UN WPP) |
bundled |
187 |
| 11 |
population_us |
US population by age (Census) |
fetched |
51 |
| 12 |
prem |
Contact matrices (Prem 2021 + 2017) |
bundled |
187 |
| 13 |
prem2017 |
Contact matrices (Prem 2017) |
fetched |
152 |
| 14 |
prem2021 |
Contact matrices (Prem 2021) |
bundled |
177 |
Bundled
The demographic data (one year per decade, Medium variant), the default contact matrices (prem2021, plus prem2017 for the ten locations it lacks), the location table and the subnational node table are inside the package, about 9 MB in all. Everyone who installs a given version has exactly the same data, it works offline, and citing the version is enough to say which data an analysis used.
load() only ever reads bundled data. If you ask it for anything else, such as ssd.load('kenya', which='life_table'), it doesn’t download it; it gives an error telling you the fetch() or download() call to use instead.
Fetched
Some data this package prepares is too large to include, or needed by too few people:
- The full
prem2017 and mistry2021 contact matrices, about 3 MB each.
- The US states’ contact matrices (
mistry2021_us, 5 MB) and their populations by single year of age (population_us).
These are prepared the same way as the bundled data and published with each release of the package. fetch() downloads one the first time you use it, checks that it is the file that was published, and saves it. As with bundled data, everyone on a given version gets the same data, and you can list which locations a fetched dataset covers without downloading it.
Downloaded
Everything else comes straight from the organization that publishes it:
- Annual UN WPP series, complete life tables, other projection variants and summary indicators, from the UN’s bulk files (17–290 MB each).
- Administrative boundaries with geometry, from UNOCHA, geoBoundaries or GADM (tens of megabytes per country at admin level 2).
- Gridded population rasters from WorldPop (1–40 MB per country-year), and the subnational populations download_nodes() computes from them.
These are also saved after the first use, along with a note of where and when each was downloaded. Since the files belong to their publishers, a publisher that updates one changes what a fresh download gives you; Provenance and reproducibility covers how to keep an analysis reproducible anyway.
The cache
Fetched and downloaded files are saved in one folder, the cache:
PosixPath('/home/runner/.cache/starsim_data')
ssd.cache_info() # what is cached, its source, and when it was retrieved
ssd.clear_cache() # or clear_cache('wpp/*')
To put it somewhere else, set ssd.options.cache_dir or the STARSIM_DATA_CACHE_DIR environment variable.
Which kind something is
The split is by size and how widely the data is needed, not by subject. The same UN WPP 2024 release is both bundled and downloaded: five series at one year per decade take 4 MB and are bundled, while the complete annual life tables take 290 MB and are downloaded. Likewise, the contact matrices most models should use are bundled (see Datasets), and the others are fetched.
|
requirement |
call |
access |
cost |
| 0 |
Age structure, mortality, births, fertility, c... |
load() |
bundled |
instant |
| 1 |
Positions, areas and separations of administra... |
load_nodes(), distances() |
bundled |
instant |
| 2 |
The other contact matrices (prem2017, mistry2021) |
fetch() |
fetched |
3 MB each, once |
| 3 |
US states: contact matrices and populations |
fetch() |
fetched |
5 MB, once |
| 4 |
Annual rates, life tables, other variants |
download() |
downloaded |
17-290 MB, once |
| 5 |
Polygon geometry, for mapping or spatial joins |
download_shapes() |
downloaded |
1-50 MB per country |
| 6 |
Subnational population estimates |
download_nodes() |
downloaded |
1-40 MB per country |
An analysis that only uses the first two rows needs no internet connection at all, and the package version is enough to say exactly which data it used.