Skip to main content

cache

cache

Download cache for data that is too big to bundle.

Tier-0 data ships inside the wheel and load() never touches the network. Anything larger – the stores behind fetch(), and the UN WPP bulk files and spatial sources behind download() and the other download_* functions – is downloaded once into a local cache and reused from then on.

Every cached file gets a provenance sidecar (<file>.provenance.json) recording the URL it came from, when it was downloaded, its size and its SHA-256. That is laser-init’s idea, and it is the right one for cached files: unlike bundled data, where provenance is written into the HDF5 itself, a cache entry is an ordinary file on disk and a sidecar is the only place the lineage can live.

import starsim_data as ssd
ssd.cache_dir()      # where things are cached
ssd.cache_info()     # what is cached, from where, and when it was downloaded
ssd.clear_cache()    # delete it all

Functions

Name Description
cache_dir Return the directory where downloaded data is cached.
cache_info Return a dataframe of what is currently cached: file, size, source URL and date.
cache_provenance Return the recorded provenance for a cached file, or None if there is none.
cached_download Download url into the cache, or return the existing path if already cached.
clear_cache Delete cached downloads. Bundled data is never touched.

cache_dir

cache.cache_dir(subdir=None, create=False)

Return the directory where downloaded data is cached.

Resolved at call time from ssd.options.cache_dir, falling back to a platform default (~/.cache/starsim_data on Linux, ~/Library/Caches/starsim_data on macOS, %LOCALAPPDATA%\starsim_data\Cache on Windows). Nothing is created unless create=True, so merely asking where the cache is has no side effects.

Parameters

Name Type Description Default
subdir str optional subdirectory within the cache, e.g. 'wpp'. None
create bool whether to create the directory if it does not exist. False

Returns

Name Type Description
pathlib.Path: the cache directory.

Example

import starsim_data as ssd
ssd.cache_dir().name # Returns 'starsim_data'

cache_info

cache.cache_info()

Return a dataframe of what is currently cached: file, size, source URL and date.

Returns

Name Type Description
sciris.dataframe: with columns [file, size, bytes, url, downloaded], empty if
nothing is cached.

Example

import starsim_data as ssd
ssd.cache_info()

cache_provenance

cache.cache_provenance(path)

Return the recorded provenance for a cached file, or None if there is none.

cached_download

cache.cached_download(
    url,
    filename=None,
    subdir=None,
    refresh=False,
    verbose=None,
)

Download url into the cache, or return the existing path if already cached.

Parameters

Name Type Description Default
url str the URL to fetch. required
filename str name to save as (default: the last path component of the URL). None
subdir str subdirectory of the cache to put it in, e.g. 'wpp'. None
refresh bool re-download even if a cached copy exists. False
verbose bool report progress (default: ssd.options.verbose). None

Returns

Name Type Description
pathlib.Path: the path to the cached file.

clear_cache

cache.clear_cache(pattern=None, verbose=None)

Delete cached downloads. Bundled data is never touched.

Parameters

Name Type Description Default
pattern str a glob matched against each cached file’s path relative to the cache root, e.g. 'wpp/*'. If None, the whole cache is removed. None
verbose bool report what was deleted (default: ssd.options.verbose). None

Returns

Name Type Description
list the paths that were deleted.

Example

import starsim_data as ssd
ssd.clear_cache('wpp/*')