# Architecture

🧠5% 🤖95%

This page describes how the package is organized internally: what the modules do, how a call to [load()](../api/loaders.md#starsim_data.loaders.load) is dispatched, how data is stored on disk, and where the extension points are. It is intended for people modifying the package or adding data to it, rather than for people using it.

## Modules

|     | module         | role                                                 |
|-----|----------------|------------------------------------------------------|
| 0   | settings.py    | The \`options\` object: cache location and verbo...  |
| 1   | registry.py    | What datasets exist, how each is shaped, and t...    |
| 2   | locations.py   | Resolving location strings; administrative-lev...    |
| 3   | loaders.py     | Reading bundled stores: \`load\`, \`list_datasets... |
| 4   | cache.py       | The download cache and its provenance sidecars       |
| 5   | fetching.py    | Stores published with the releases: \`fetch\`        |
| 6   | downloading.py | UN WPP bulk files: \`download\`                      |
| 7   | nodes.py       | The node table and \`distances\`                     |
| 8   | shapes.py      | Boundary sources and \`download_shapes\` (needs ...  |
| 9   | rasters.py     | WorldPop rasters and \`download_nodes\` (needs \[... |
| 10  | plotting.py    | Validation figures and the choropleth                |

Import order matters, because two modules register themselves when they are imported: [loaders](../api/loaders.md#starsim_data.loaders) adds its layout shapers and [downloading](../api/downloading.md#starsim_data.downloading) adds its tier-2 datasets, both of which require [registry](../api/registry.md#starsim_data.registry) to exist first. The order is fixed in `__init__.py`.

## Import conventions

Modules follow Starsim's convention: each imports the package under its own alias and reaches everything through it.

``` python
import starsim_data as ssd

def load_nodes(location, level=0):
    return ssd.load(location, which='nodes', level=level)
```

`from` imports are confined to `__init__.py`, where the public namespace is assembled. Within a module, `ssd.load` and `ssd.registry.datasets()` resolve at call time, so modules can refer to each other freely.

## The registry

The registry holds two tables.

`DATASETS` maps a dataset name to a descriptor recording its layout, tier, administrative levels, license, available revisions, location count and where its store lives.

``` python
import starsim_data as ssd

ssd.registry.dataset('deaths')
```

    Dataset(name='deaths', layout='mortality', tier=0, levels=(0,), sources=(Source(store='demographics-2024.h5', group='deaths', revision='2024', url='', sha256=''),), license='CC BY 3.0 IGO', n_locations=187, description='Mortality by age and sex (UN WPP)')

Descriptors for bundled data are discovered once, by opening each `.h5` file in the data directory and reading its root attributes. The result is cached; `registry.refresh()` discards the cache, which is needed only if the files change within a running process, such as after a rebuild in a test. The default contact matrices, `prem`, are described at the same time, from `prem2021.h5` and the ten matrices in `prem/prem2017.h5`; their descriptor has a `loader` that reads whichever store has the location.

`LAYOUTS` maps a layout name to the function that turns a stored array into a dataframe.

``` python
sorted(ssd.registry.LAYOUTS)
```

    ['adjacency',
     'crude',
     'fertility',
     'matrix',
     'mortality',
     'nodes',
     'population']

Adding a new family of data is a shaper plus a registry entry.

## How loading is dispatched

    load(location, which, kind, revision, level)
      │
      ├─ registry.dataset(which)          → the descriptor, or an error with suggestions
      ├─ descriptor.loader?               → delegate (e.g. `prem`, which picks prem2021 or prem2017)
      ├─ descriptor.tier > 0?             → refuse, and point at fetch() or download()
      ├─ descriptor.source(revision)      → which file, and which group inside it
      ├─ h5py.File(...)                   → open the store
      ├─ resolve(location, valid=...)     → the canonical key, scoped to this dataset
      ├─ registry.get_layout(spec.layout) → the shaper for this layout
      └─ shaper(group, key, spec, opts)   → the dataframe, plus provenance in .attrs

The `opts` dictionary carries the arguments that apply to some layouts and not others -- `kind` for contact matrices, [level](../api/registry.md#starsim_data.registry.Dataset.level) for the node table -- so a new layout can introduce its own without changing any existing shaper's signature. An argument applied to a layout that has no use for it is rejected rather than ignored:

``` python
ssd.load('Kenya', which='deaths', level=1)
```

    ---------------------------------------------------------------------------
    ValueError                                Traceback (most recent call last)
    Cell In[4], line 1
    ----> 1 ssd.load('Kenya', which='deaths', level=1)

    File ~/work/starsim_data/starsim_data/starsim_data/loaders.py:287, in load(location, which, kind, revision, level)
        285     return ds.loader(location, kind=kind, revision=revision, level=level)
        286 if (level is not None) and (ds.layout != 'nodes'):
    --> 287     raise ValueError(f'level={level!r} does not apply to {which!r}; only the node table is addressed by administrative level')
        288 if ds.tier > 0: # `load` never touches the network; say so rather than quietly downloading
        289     raise ValueError(f'{which!r} is not bundled; use {ds.function}({location!r}, which={which!r}) to download it')

    ValueError: level=1 does not apply to 'deaths'; only the node table is addressed by administrative level

## Storage

Bundled data is stored as HDF5, one store per family, with a `spec` attribute describing the layout and a [provenance](../api/loaders.md#starsim_data.loaders.provenance) attribute describing the source. The stores are self-describing: the loader reads how to interpret a file from the file itself.

|  | file | layout | structure |
|----|----|----|----|
| 0 | prem2021.h5, prem/prem2017.h5 (and, fetched, p... | matrix | /\<Location\>/\<setting\> → a square float32 array |
| 1 | fetched.json | -- | the stores that are fetched: URL, SHA-256, loc... |
| 2 | demographics-\<revision\>.h5 | crude, mortality, population, fertility | /\<series\>/\<Location\> → a (year\[, sex\]\[, age\]) ... |
| 3 | population_us.h5 (fetched) | population | /\<Country:Unit\> → a (year, sex, age) array |
| 4 | nodes.h5 | nodes | /\<Location\> → a compound array, one row per unit |
| 5 | locations.csv | -- | the canonical location table, read with the cs... |

Demographic stores are named by revision, so a new UN release adds a file rather than replacing one. The registry gathers the revisions of a series into a single descriptor, and `Dataset.source(revision)` selects among them, defaulting to the most recent.

## Tiers, in code

Internally, whether a dataset is bundled is its *tier*, a field on the descriptor: 0 for bundled, 1 for fetched and 2 for downloaded (the descriptor's [access](../api/registry.md#starsim_data.registry.Dataset.access) property gives the same in the words used in [Bundled, fetched and downloaded data](access.md), and its [function](../api/registry.md#starsim_data.registry.Dataset.function) property the function for each). [load()](../api/loaders.md#starsim_data.loaders.load) enforces it:

``` python
ssd.load('kenya', which='life_table')
```

    ---------------------------------------------------------------------------
    ValueError                                Traceback (most recent call last)
    Cell In[6], line 1
    ----> 1 ssd.load('kenya', which='life_table')

    File ~/work/starsim_data/starsim_data/starsim_data/loaders.py:289, in load(location, which, kind, revision, level)
        287     raise ValueError(f'level={level!r} does not apply to {which!r}; only the node table is addressed by administrative level')
        288 if ds.tier > 0: # `load` never touches the network; say so rather than quietly downloading
    --> 289     raise ValueError(f'{which!r} is not bundled; use {ds.function}({location!r}, which={which!r}) to download it')
        290 return _read(which, location, kind=kind, revision=revision, level=level)

    ValueError: 'life_table' is not bundled; use download('kenya', which='life_table') to download it

Tier-1 datasets appear in the registry despite having no bundled store, which is what lets [load()](../api/loaders.md#starsim_data.loaders.load) redirect to [fetch()](../api/fetching.md#starsim_data.fetching.fetch) or [download()](../api/downloading.md#starsim_data.downloading.download) by name rather than fail to find a file.

## Extension points

Three public hooks are provided, all of which are used internally by the package itself.

**A new layout.** Register a function that turns a stored array into a dataframe:

``` python
def shape_my_layout(group, key, spec, opts):
    return sc.dataframe(group[key][:])

ssd.register_layout('my_layout', shape_my_layout)
```

**A new dataset.** Register a descriptor. Giving it a `loader` causes [load()](../api/loaders.md#starsim_data.loaders.load) to delegate to it, which is the route for data this package does not ship:

``` python
ds = ssd.registry.Dataset(name='my_dataset', layout='crude', tier=2,
                          description='Something from elsewhere',
                          loader=lambda location, **kw: f'data for {location}')
ssd.register_dataset(ds)
ssd.load('Kenya', which='my_dataset')
```

    'data for Kenya'

    Dataset(name='my_dataset', layout='crude', tier=2, levels=(0,), sources=(), license='', n_locations=0, description='Something from elsewhere')

**A new plotter.** `plotting.PLOTTERS` maps a layout to a drawing function, and can be assigned into.

All three hooks are functions rather than dictionaries you mutate, so a downstream package can extend the registry without reaching into it.

## Configuration and side effects

Importing the package creates no directories, opens no files and emits no warnings. Settings are read from the environment at import, but acted on only when a function needs them: the cache directory is resolved at call time and created when something is first written to it.

``` python
ssd.options.help()
```

       cache_dir = None                         # Where downloaded (non-bundled) data is cached. None means a platform-appropriate default; see cache_dir().
         verbose = True                         # Whether to report progress while downloading.
          pandas = False                        # Whether to return plain pandas DataFrames rather than Sciris dataframes.

There is no logging configuration; handler setup is left to the application.

## Testing

The test suite is divided by what it requires.

|  | marker | requires | note |
|----|----|----|----|
| 0 | (none) | Nothing but the bundled data | The majority; hermetic and fast |
| 1 | network | Internet access | Deselected by default in CI; set STARSIM_DATA\_... |
| 2 | slow | More than a second or so | Chiefly the 145 MB life-table downloads |
| 3 | spatial | The \[spatial\] extra | Skipped automatically when it is absent |

Continuous integration also rebuilds every store from `raw_data/` and re-runs the suite against the result, so the committed artifacts cannot drift from their sources. The node table has a separate, manually triggered workflow, since rebuilding it downloads roughly 1.4 GB.
