Skip to main content

registry

registry

The dataset registry: what datasets exist, where they live, and how to shape them.

Two tables live here.

DATASETS maps a dataset name to a Dataset descriptor – its layout, tier, admin level, license, available UN WPP revisions and location count. The tier says how the data is got: 0 bundled (load()), 1 fetched from this package’s releases (fetch()), 2 downloaded from its original publisher (download()). Bundled datasets are discovered once by reading the HDF5 stores in data/. Stores that are built the same way but kept out of the wheel (tier 1, fetched on demand by fetch()) are described by the manifest data/fetched.json, which scripts/build_hdf5.py writes. Anything else can be added with register_dataset().

LAYOUTS maps a layout name to the function that turns a stored array into a dataframe (matrix, crude, mortality, population, fertility, nodes). This replaces dispatching on spec['layout'] through an if-chain, and means a new data family is a new shaper plus a registry entry rather than another branch inside load().

Both tables have a public registration hook, which is the point: laser-init’s registry is module-level dict literals with no way to extend them, which its own integration notes identify as a blocker to downstream reuse.

import starsim_data as ssd
ssd.list_datasets(detailed='full') # the registry as a dataframe
ssd.registry.dataset('deaths')     # one descriptor

Classes

Name Description
Dataset What a dataset is: its shape, its provenance and where to get it.
Source Where one revision of a dataset physically lives.

Dataset

registry.Dataset(
    name,
    layout,
    tier=0,
    levels=(0,),
    sources=(),
    license='',
    n_locations=0,
    description='',
    loader=None,
    locations=(),
    provenance=dict(),
)

What a dataset is: its shape, its provenance and where to get it.

Attributes

Name Type Description
name str the dataset name, as passed to load(which=...).
layout str how the values are stored; see LAYOUTS.
tier int 0 if the dataset is bundled in the wheel, 1 if it is a store this package publishes and fetch() downloads, 2 if download() gets it from its original publisher (see access and function for the same in words).
levels tuple the administrative levels covered; (0,) for national data.
sources tuple Source records, oldest revision first.
license str the upstream data license, e.g. 'CC BY 3.0 IGO'.
n_locations int how many locations the dataset covers.
description str a one-line summary.
loader callable if set, load() delegates to it instead of reading a bundled store – the hook for registering a dataset this package does not ship.
locations tuple for a store that is fetched, the locations it holds, so that they can be listed without downloading it.
provenance dict for a store that is fetched, the provenance recorded in it.

Methods

Name Description
source Return the Source for revision, defaulting to the newest available.

source

registry.Dataset.source(revision=None)

Return the Source for revision, defaulting to the newest available.

Raises
Type Description
ValueError if revision is given for a dataset that is not versioned by revision, or names a revision that is not available.

Source

registry.Source(store, group='', revision='', url='', sha256='')

Where one revision of a dataset physically lives.

Attributes

Name Description
bundled Whether the store ships inside the wheel (rather than being fetched).

Functions

Name Description
data_dir Return the directory holding the bundled data files.
dataset Return the Dataset descriptor for name, which may be an alias (see ALIASES).
datasets Return every known dataset as {name: Dataset}.
get_layout Return the shaper registered for a layout.
refresh Forget the discovered datasets, so the stores are re-read on next use.
register_dataset Add a dataset to the registry.
register_layout Register a function that turns a stored array into a dataframe.
unregister_dataset Remove a previously registered dataset. Bundled datasets cannot be removed.

data_dir

registry.data_dir()

Return the directory holding the bundled data files.

The single place the bundled data is located, so that a test (or a rebuild) can point the whole package at a different directory by patching this one function – followed by refresh(), since discovery is cached.

dataset

registry.dataset(name)

Return the Dataset descriptor for name, which may be an alias (see ALIASES).

Raises

Type Description
ValueError if the dataset is unknown; the message lists close matches.

datasets

registry.datasets(tier=None)

Return every known dataset as {name: Dataset}.

Parameters

Name Type Description Default
tier int if given, only datasets in this tier (0 bundled, 1 fetched, 2 downloaded). None

Returns

Name Type Description
sciris.objdict: name -> Dataset, sorted by name.

get_layout

registry.get_layout(name)

Return the shaper registered for a layout.

Raises

Type Description
ValueError if the layout is unknown.

refresh

registry.refresh()

Forget the discovered datasets, so the stores are re-read on next use.

Only needed if the bundled data changes within a running process, e.g. after a rebuild in a test.

register_dataset

registry.register_dataset(dataset, overwrite=False)

Add a dataset to the registry.

The hook for data this package does not ship: give the descriptor a loader, and load(which=<name>) will call it as loader(location, **kwargs).

Parameters

Name Type Description Default
dataset Dataset the descriptor to register. required
overwrite bool whether to replace an existing dataset of the same name. False

Example

import starsim_data as ssd
ds = ssd.registry.Dataset(name='mydata', layout='crude', tier=2,
                          loader=lambda location, **kw: location)
ssd.registry.register_dataset(ds)
ssd.load('Kenya', which='mydata') # Returns 'Kenya'
ssd.registry.unregister_dataset('mydata')

register_layout

registry.register_layout(name, shaper, overwrite=False)

Register a function that turns a stored array into a dataframe.

Parameters

Name Type Description Default
name str the layout name, matching spec['layout'] in the store. required
shaper callable called as shaper(group, key, spec, opts) and returning a sciris.dataframe. opts carries the per-call arguments that apply to some layouts but not others – kind for contact matrices, level for nodes. required
overwrite bool whether to replace an existing layout of the same name. False

unregister_dataset

registry.unregister_dataset(name)

Remove a previously registered dataset. Bundled datasets cannot be removed.