# Bundled, fetched and downloaded data

🧠15% 🤖85%

Every dataset in `starsim_data` is one of three kinds, and each kind has its own function, so you can tell from the call where the data comes from:

|  | Comes from | How you get it | Internet needed | Same for everyone on a version? |
|----|----|----|----|----|
| **Bundled** | Inside the package | [load()](../api/loaders.md#starsim_data.loaders.load), [load_nodes()](../api/nodes.md#starsim_data.nodes.load_nodes), [distances()](../api/nodes.md#starsim_data.nodes.distances), [adjacency()](../api/nodes.md#starsim_data.nodes.adjacency) | Never | Yes |
| **Fetched** | This package's releases | [fetch()](../api/fetching.md#starsim_data.fetching.fetch) | First use only | Yes |
| **Downloaded** | The original publisher | [download()](../api/downloading.md#starsim_data.downloading.download), [download_shapes()](../api/shapes.md#starsim_data.shapes.download_shapes), [download_nodes()](../api/rasters.md#starsim_data.rasters.download_nodes), [download_raster()](../api/rasters.md#starsim_data.rasters.download_raster) | First use only | Unless the publisher changes the file |

Fetched and downloaded data are both saved on your computer after the first use, so they need the internet only once. The difference is where they come from: [fetch()](../api/fetching.md#starsim_data.fetching.fetch) gets files from this package's own releases, while [download()](../api/downloading.md#starsim_data.downloading.download) goes to the UN, WorldPop and the boundary publishers. Their servers are outside this package's control, and may be slow, change their files or be down. Calling [fetch()](../api/fetching.md#starsim_data.fetching.fetch) for data that isn't packaged still works, but warns that it is downloading from an outside source.

The [access](../api/registry.md#starsim_data.registry.Dataset.access) column of [list_datasets()](../api/loaders.md#starsim_data.loaders.list_datasets) says which kind each dataset is:

``` python
import starsim_data as ssd

ssd.list_datasets(detailed=True)
```

|     | dataset       | description                          | access     | locations |
|-----|---------------|--------------------------------------|------------|-----------|
| 0   | adjacency     | Admin units that share a border      | bundled    | 184       |
| 1   | births        | Crude birth rate (UN WPP)            | bundled    | 187       |
| 2   | deaths        | Mortality by age and sex (UN WPP)    | bundled    | 187       |
| 3   | fertility     | Fertility by age of mother (UN WPP)  | bundled    | 187       |
| 4   | indicators    | All demographic indicators (UN WPP)  | downloaded | 0         |
| 5   | life_table    | Abridged life tables (UN WPP)        | downloaded | 0         |
| 6   | migration     | Net migration rate (UN WPP)          | bundled    | 187       |
| 7   | mistry2021    | Contact matrices (Mistry 2021)       | fetched    | 36        |
| 8   | mistry2021_us | US contact matrices (Mistry 2021)    | fetched    | 51        |
| 9   | nodes         | Admin units: centroids, areas, codes | bundled    | 184       |
| 10  | population    | Population by age and sex (UN WPP)   | bundled    | 187       |
| 11  | population_us | US population by age (Census)        | fetched    | 51        |
| 12  | prem          | Contact matrices (Prem 2021 + 2017)  | bundled    | 187       |
| 13  | prem2017      | Contact matrices (Prem 2017)         | fetched    | 152       |
| 14  | prem2021      | Contact matrices (Prem 2021)         | bundled    | 177       |

## Bundled

The demographic data (one year per decade, Medium variant), the default contact matrices (`prem2021`, plus `prem2017` for the ten locations it lacks), the location table and the subnational node table are inside the package, about 9 MB in all. Everyone who installs a given version has exactly the same data, it works offline, and citing the version is enough to say which data an analysis used.

[load()](../api/loaders.md#starsim_data.loaders.load) only ever reads bundled data. If you ask it for anything else, such as `ssd.load('kenya', which='life_table')`, it doesn't download it; it gives an error telling you the [fetch()](../api/fetching.md#starsim_data.fetching.fetch) or [download()](../api/downloading.md#starsim_data.downloading.download) call to use instead.

## Fetched

Some data this package prepares is too large to include, or needed by too few people:

- The full `prem2017` and `mistry2021` contact matrices, about 3 MB each.
- The US states' contact matrices (`mistry2021_us`, 5 MB) and their populations by single year of age (`population_us`).

These are prepared the same way as the bundled data and published with each release of the package. [fetch()](../api/fetching.md#starsim_data.fetching.fetch) downloads one the first time you use it, checks that it is the file that was published, and saves it. As with bundled data, everyone on a given version gets the same data, and you can list which locations a fetched dataset covers without downloading it.

## Downloaded

Everything else comes straight from the organization that publishes it:

- Annual UN WPP series, complete life tables, other projection variants and summary indicators, from the UN's bulk files (17-290 MB each).
- Administrative boundaries with geometry, from UNOCHA, geoBoundaries or GADM (tens of megabytes per country at admin level 2).
- Gridded population rasters from WorldPop (1-40 MB per country-year), and the subnational populations [download_nodes()](../api/rasters.md#starsim_data.rasters.download_nodes) computes from them.

These are also saved after the first use, along with a note of where and when each was downloaded. Since the files belong to their publishers, a publisher that updates one changes what a fresh download gives you; [Provenance and reproducibility](provenance.md) covers how to keep an analysis reproducible anyway.

## The cache

Fetched and downloaded files are saved in one folder, the cache:

``` python
ssd.cache_dir()
```

    PosixPath('/home/runner/.cache/starsim_data')

``` python
ssd.cache_info()      # what is cached, its source, and when it was retrieved
ssd.clear_cache()     # or clear_cache('wpp/*')
```

To put it somewhere else, set `ssd.options.cache_dir` or the `STARSIM_DATA_CACHE_DIR` environment variable.

## Which kind something is

The split is by size and how widely the data is needed, not by subject. The same UN WPP 2024 release is both bundled and downloaded: five series at one year per decade take 4 MB and are bundled, while the complete annual life tables take 290 MB and are downloaded. Likewise, the contact matrices most models should use are bundled (see [Datasets](datasets.md#which-source)), and the others are fetched.

|  | requirement | call | access | cost |
|----|----|----|----|----|
| 0 | Age structure, mortality, births, fertility, c... | load() | bundled | instant |
| 1 | Positions, areas and separations of administra... | load_nodes(), distances() | bundled | instant |
| 2 | The other contact matrices (prem2017, mistry2021) | fetch() | fetched | 3 MB each, once |
| 3 | US states: contact matrices and populations | fetch() | fetched | 5 MB, once |
| 4 | Annual rates, life tables, other variants | download() | downloaded | 17-290 MB, once |
| 5 | Polygon geometry, for mapping or spatial joins | download_shapes() | downloaded | 1-50 MB per country |
| 6 | Subnational population estimates | download_nodes() | downloaded | 1-40 MB per country |

An analysis that only uses the first two rows needs no internet connection at all, and the package version is enough to say exactly which data it used.
