# registry

[registry](../api/registry.md#starsim_data.registry)

The dataset registry: what datasets exist, where they live, and how to shape them.

Two tables live here.

`DATASETS` maps a dataset name to a [Dataset](../api/registry.md#starsim_data.registry.Dataset) descriptor - its layout, tier, admin level, license, available UN WPP revisions and location count. The tier says how the data is got: 0 bundled ([load()](../api/loaders.md#starsim_data.loaders.load)), 1 fetched from this package's releases ([fetch()](../api/fetching.md#starsim_data.fetching.fetch)), 2 downloaded from its original publisher ([download()](../api/downloading.md#starsim_data.downloading.download)). Bundled datasets are discovered once by reading the HDF5 stores in `data/`. Stores that are built the same way but kept out of the wheel (tier 1, fetched on demand by [fetch()](../api/fetching.md#starsim_data.fetching.fetch)) are described by the manifest `data/fetched.json`, which `scripts/build_hdf5.py` writes. Anything else can be added with [register_dataset()](../api/registry.md#starsim_data.registry.register_dataset).

`LAYOUTS` maps a layout name to the function that turns a stored array into a dataframe (`matrix`, `crude`, `mortality`, `population`, `fertility`, [nodes](../api/nodes.md#starsim_data.nodes)). This replaces dispatching on `spec['layout']` through an if-chain, and means a new data family is a new shaper plus a registry entry rather than another branch inside [load()](../api/loaders.md#starsim_data.loaders.load).

Both tables have a public registration hook, which is the point: `laser-init`'s registry is module-level dict literals with no way to extend them, which its own integration notes identify as a blocker to downstream reuse.

``` python
import starsim_data as ssd
ssd.list_datasets(detailed='full') # the registry as a dataframe
ssd.registry.dataset('deaths')     # one descriptor
```

## Classes

| Name | Description |
|----|----|
| [Dataset](#starsim_data.registry.Dataset) | What a dataset is: its shape, its provenance and where to get it. |
| [Source](#starsim_data.registry.Source) | Where one revision of a dataset physically lives. |

### Dataset

``` python
registry.Dataset(
    name,
    layout,
    tier=0,
    levels=(0,),
    sources=(),
    license='',
    n_locations=0,
    description='',
    loader=None,
    locations=(),
    provenance=dict(),
)
```

What a dataset is: its shape, its provenance and where to get it.

#### Attributes

| Name | Type | Description |
|----|----|----|
| name | str | the dataset name, as passed to `load(which=...)`. |
| layout | str | how the values are stored; see `LAYOUTS`. |
| tier | int | 0 if the dataset is bundled in the wheel, 1 if it is a store this package publishes and [fetch()](../api/fetching.md#starsim_data.fetching.fetch) downloads, 2 if [download()](../api/downloading.md#starsim_data.downloading.download) gets it from its original publisher (see [access](../api/registry.md#starsim_data.registry.Dataset.access) and [function](../api/registry.md#starsim_data.registry.Dataset.function) for the same in words). |
| levels | tuple | the administrative levels covered; `(0,)` for national data. |
| sources | tuple | [Source](../api/registry.md#starsim_data.registry.Source) records, oldest revision first. |
| license | str | the upstream data license, e.g. `'CC BY 3.0 IGO'`. |
| n_locations | int | how many locations the dataset covers. |
| description | str | a one-line summary. |
| loader | callable | if set, [load()](../api/loaders.md#starsim_data.loaders.load) delegates to it instead of reading a bundled store - the hook for registering a dataset this package does not ship. |
| locations | tuple | for a store that is fetched, the locations it holds, so that they can be listed without downloading it. |
| provenance | dict | for a store that is fetched, the provenance recorded in it. |

#### Methods

| Name | Description |
|----|----|
| [source](#starsim_data.registry.Dataset.source) | Return the [Source](../api/registry.md#starsim_data.registry.Source) for `revision`, defaulting to the newest available. |

##### source

``` python
registry.Dataset.source(revision=None)
```

Return the [Source](../api/registry.md#starsim_data.registry.Source) for `revision`, defaulting to the newest available.

###### Raises

| Type | Description |
|----|----|
| ValueError | if `revision` is given for a dataset that is not versioned by revision, or names a revision that is not available. |

### Source

``` python
registry.Source(store, group='', revision='', url='', sha256='')
```

Where one revision of a dataset physically lives.

#### Attributes

| Name | Description |
|----|----|
| [bundled](#starsim_data.registry.Source.bundled) | Whether the store ships inside the wheel (rather than being fetched). |

## Functions

| Name | Description |
|----|----|
| [data_dir](#starsim_data.registry.data_dir) | Return the directory holding the bundled data files. |
| [dataset](#starsim_data.registry.dataset) | Return the [Dataset](../api/registry.md#starsim_data.registry.Dataset) descriptor for `name`, which may be an alias (see `ALIASES`). |
| [datasets](#starsim_data.registry.datasets) | Return every known dataset as `{name: Dataset}`. |
| [get_layout](#starsim_data.registry.get_layout) | Return the shaper registered for a layout. |
| [refresh](#starsim_data.registry.refresh) | Forget the discovered datasets, so the stores are re-read on next use. |
| [register_dataset](#starsim_data.registry.register_dataset) | Add a dataset to the registry. |
| [register_layout](#starsim_data.registry.register_layout) | Register a function that turns a stored array into a dataframe. |
| [unregister_dataset](#starsim_data.registry.unregister_dataset) | Remove a previously registered dataset. Bundled datasets cannot be removed. |

### data_dir

``` python
registry.data_dir()
```

Return the directory holding the bundled data files.

The single place the bundled data is located, so that a test (or a rebuild) can point the whole package at a different directory by patching this one function - followed by [refresh()](../api/registry.md#starsim_data.registry.refresh), since discovery is cached.

### dataset

``` python
registry.dataset(name)
```

Return the [Dataset](../api/registry.md#starsim_data.registry.Dataset) descriptor for `name`, which may be an alias (see `ALIASES`).

#### Raises

| Type       | Description                                                 |
|------------|-------------------------------------------------------------|
| ValueError | if the dataset is unknown; the message lists close matches. |

### datasets

``` python
registry.datasets(tier=None)
```

Return every known dataset as `{name: Dataset}`.

#### Parameters

| Name | Type | Description | Default |
|----|----|----|----|
| tier | int | if given, only datasets in this tier (0 bundled, 1 fetched, 2 downloaded). | `None` |

#### Returns

| Name | Type | Description |
|----|----|----|
|  |  | sciris.objdict: name -\> [Dataset](../api/registry.md#starsim_data.registry.Dataset), sorted by name. |

### get_layout

``` python
registry.get_layout(name)
```

Return the shaper registered for a layout.

#### Raises

| Type       | Description               |
|------------|---------------------------|
| ValueError | if the layout is unknown. |

### refresh

``` python
registry.refresh()
```

Forget the discovered datasets, so the stores are re-read on next use.

Only needed if the bundled data changes within a running process, e.g. after a rebuild in a test.

### register_dataset

``` python
registry.register_dataset(dataset, overwrite=False)
```

Add a dataset to the registry.

The hook for data this package does not ship: give the descriptor a `loader`, and `load(which=<name>)` will call it as `loader(location, **kwargs)`.

#### Parameters

| Name | Type | Description | Default |
|----|----|----|----|
| dataset | [Dataset](../api/registry.md#starsim_data.registry.Dataset) | the descriptor to register. | *required* |
| overwrite | bool | whether to replace an existing dataset of the same name. | `False` |

#### Example

``` python
import starsim_data as ssd
ds = ssd.registry.Dataset(name='mydata', layout='crude', tier=2,
                          loader=lambda location, **kw: location)
ssd.registry.register_dataset(ds)
ssd.load('Kenya', which='mydata') # Returns 'Kenya'
ssd.registry.unregister_dataset('mydata')
```

### register_layout

``` python
registry.register_layout(name, shaper, overwrite=False)
```

Register a function that turns a stored array into a dataframe.

#### Parameters

| Name | Type | Description | Default |
|----|----|----|----|
| name | str | the layout name, matching `spec['layout']` in the store. | *required* |
| shaper | callable | called as `shaper(group, key, spec, opts)` and returning a `sciris.dataframe`. `opts` carries the per-call arguments that apply to some layouts but not others - `kind` for contact matrices, [level](../api/registry.md#starsim_data.registry.Dataset.level) for nodes. | *required* |
| overwrite | bool | whether to replace an existing layout of the same name. | `False` |

### unregister_dataset

``` python
registry.unregister_dataset(name)
```

Remove a previously registered dataset. Bundled datasets cannot be removed.
