registry
The dataset registry: what datasets exist, where they live, and how to shape them.
Two tables live here.
DATASETS maps a dataset name to a Dataset descriptor – its layout, tier, admin level, license, available UN WPP revisions and location count. The tier says how the data is got: 0 bundled (load()), 1 fetched from this package’s releases (fetch()), 2 downloaded from its original publisher (download()). Bundled datasets are discovered once by reading the HDF5 stores in data/. Stores that are built the same way but kept out of the wheel (tier 1, fetched on demand by fetch()) are described by the manifest data/fetched.json, which scripts/build_hdf5.py writes. Anything else can be added with register_dataset().
LAYOUTS maps a layout name to the function that turns a stored array into a dataframe (matrix, crude, mortality, population, fertility, nodes). This replaces dispatching on spec['layout'] through an if-chain, and means a new data family is a new shaper plus a registry entry rather than another branch inside load().
Both tables have a public registration hook, which is the point: laser-init’s registry is module-level dict literals with no way to extend them, which its own integration notes identify as a blocker to downstream reuse.
import starsim_data as ssd
ssd.list_datasets(detailed='full') # the registry as a dataframe
ssd.registry.dataset('deaths') # one descriptorClasses
| Name | Description |
|---|---|
| Dataset | What a dataset is: its shape, its provenance and where to get it. |
| Source | Where one revision of a dataset physically lives. |
Dataset
registry.Dataset(
name,
layout,
tier=0,
levels=(0,),
sources=(),
license='',
n_locations=0,
description='',
loader=None,
locations=(),
provenance=dict(),
)What a dataset is: its shape, its provenance and where to get it.
Attributes
| Name | Type | Description |
|---|---|---|
| name | str | the dataset name, as passed to load(which=...). |
| layout | str | how the values are stored; see LAYOUTS. |
| tier | int | 0 if the dataset is bundled in the wheel, 1 if it is a store this package publishes and fetch() downloads, 2 if download() gets it from its original publisher (see access and function for the same in words). |
| levels | tuple | the administrative levels covered; (0,) for national data. |
| sources | tuple | Source records, oldest revision first. |
| license | str | the upstream data license, e.g. 'CC BY 3.0 IGO'. |
| n_locations | int | how many locations the dataset covers. |
| description | str | a one-line summary. |
| loader | callable | if set, load() delegates to it instead of reading a bundled store – the hook for registering a dataset this package does not ship. |
| locations | tuple | for a store that is fetched, the locations it holds, so that they can be listed without downloading it. |
| provenance | dict | for a store that is fetched, the provenance recorded in it. |
Methods
| Name | Description |
|---|---|
| source | Return the Source for revision, defaulting to the newest available. |
source
registry.Dataset.source(revision=None)Return the Source for revision, defaulting to the newest available.
Raises
| Type | Description |
|---|---|
| ValueError | if revision is given for a dataset that is not versioned by revision, or names a revision that is not available. |
Source
registry.Source(store, group='', revision='', url='', sha256='')Where one revision of a dataset physically lives.
Attributes
| Name | Description |
|---|---|
| bundled | Whether the store ships inside the wheel (rather than being fetched). |
Functions
| Name | Description |
|---|---|
| data_dir | Return the directory holding the bundled data files. |
| dataset | Return the Dataset descriptor for name, which may be an alias (see ALIASES). |
| datasets | Return every known dataset as {name: Dataset}. |
| get_layout | Return the shaper registered for a layout. |
| refresh | Forget the discovered datasets, so the stores are re-read on next use. |
| register_dataset | Add a dataset to the registry. |
| register_layout | Register a function that turns a stored array into a dataframe. |
| unregister_dataset | Remove a previously registered dataset. Bundled datasets cannot be removed. |
data_dir
registry.data_dir()Return the directory holding the bundled data files.
The single place the bundled data is located, so that a test (or a rebuild) can point the whole package at a different directory by patching this one function – followed by refresh(), since discovery is cached.
dataset
registry.dataset(name)Return the Dataset descriptor for name, which may be an alias (see ALIASES).
Raises
| Type | Description |
|---|---|
| ValueError | if the dataset is unknown; the message lists close matches. |
datasets
registry.datasets(tier=None)Return every known dataset as {name: Dataset}.
Parameters
| Name | Type | Description | Default |
|---|---|---|---|
| tier | int | if given, only datasets in this tier (0 bundled, 1 fetched, 2 downloaded). | None |
Returns
| Name | Type | Description |
|---|---|---|
| sciris.objdict: name -> Dataset, sorted by name. |
get_layout
registry.get_layout(name)Return the shaper registered for a layout.
Raises
| Type | Description |
|---|---|
| ValueError | if the layout is unknown. |
refresh
registry.refresh()Forget the discovered datasets, so the stores are re-read on next use.
Only needed if the bundled data changes within a running process, e.g. after a rebuild in a test.
register_dataset
registry.register_dataset(dataset, overwrite=False)Add a dataset to the registry.
The hook for data this package does not ship: give the descriptor a loader, and load(which=<name>) will call it as loader(location, **kwargs).
Parameters
| Name | Type | Description | Default |
|---|---|---|---|
| dataset | Dataset | the descriptor to register. | required |
| overwrite | bool | whether to replace an existing dataset of the same name. | False |
Example
import starsim_data as ssd
ds = ssd.registry.Dataset(name='mydata', layout='crude', tier=2,
loader=lambda location, **kw: location)
ssd.registry.register_dataset(ds)
ssd.load('Kenya', which='mydata') # Returns 'Kenya'
ssd.registry.unregister_dataset('mydata')register_layout
registry.register_layout(name, shaper, overwrite=False)Register a function that turns a stored array into a dataframe.
Parameters
| Name | Type | Description | Default |
|---|---|---|---|
| name | str | the layout name, matching spec['layout'] in the store. |
required |
| shaper | callable | called as shaper(group, key, spec, opts) and returning a sciris.dataframe. opts carries the per-call arguments that apply to some layouts but not others – kind for contact matrices, level for nodes. |
required |
| overwrite | bool | whether to replace an existing layout of the same name. | False |
unregister_dataset
registry.unregister_dataset(name)Remove a previously registered dataset. Bundled datasets cannot be removed.