Contributing
How to change starsim_data and release it. Every change goes through a pull request to main; nothing is committed to main directly.
Setting up
git clone https://github.com/starsimhub/starsim_data
cd starsim_data
pip install -e ".[dev]" # add ,spatial to rebuild the node table
pytest -m "not network" # the offline suite, as CI runs itChanging code
Make the change on a branch, with a test if it fixes or adds something, and update the prose it affects: the API reference regenerates itself, but the pages in docs/user-guide/ don’t. A new dataset or source also touches docs/user-guide/04-datasets.qmd, docs/user-guide/07-datasources.qmd and NOTICE.md (a test checks the licenses there against the data).
Building the documentation
The documentation is built with Sciris’ fork of great-docs and quartopydoc, which the docs dependency group installs, plus Quarto:
pip install -e ".[dev]" --group docs
great-docs build # -> docs/_site/
docs/render # the same, rendering pages in parallel
great-docs preview # live reload on http://localhost:3000Adding a page means adding a .qmd file to docs/user-guide/ with a title and a guide-section; there is no navigation configuration to update (a new module, though, needs adding to the quartodoc: sections in docs/great-docs.yml). docs/user-guide/assets/starsim-theme.html holds the style overrides that make the site look like the Starsim docs.
Changing data
The .h5 files are build artifacts: never edit them by hand. Edit the sources in raw_data/ and rebuild with the scripts in scripts/ (its README describes each). Then run them with --check, which is what CI runs: it rebuilds into a temporary folder and fails if the result differs from the committed files, without writing anything.
python scripts/build_hdf5.py --check
python scripts/build_demographics.py --checkThe fetched stores (in fetched/, downloaded by fetch() rather than shipped in the package) are published as assets of a GitHub release, named in RELEASE in scripts/build_hdf5.py. They are rebuilt only with python scripts/build_hdf5.py --fetched. If a store in fetched/ changes, or a new one is added, set a new tag in RELEASE (e.g. data-2027.03) and rerun the script, so that the manifest points at a release that holds the new bytes. Data releases are never edited once published.
Releasing
In the pull request, bump
__version__and__versiondate__instarsim_data/version.pyand add a section for that version toCHANGELOG.md, headed## Version X.Y.Z (date). That section becomes the release notes.Merge the pull request, then update your local
main:git switch main git pullRun the release script:
scripts/publish_releaseIt is the same whether or not the data has changed; what decides it is
RELEASEinscripts/build_hdf5.py, which names the data release the fetched stores are downloaded from (see above), and the script says up front which case it is:- Package only (the usual case): no fetched store has changed, so the release named in
RELEASEalready exists. The script checks that its files are all there and creates onlyv<version>. - New data and the package: you changed a fetched store and set a new
RELEASE, so that release doesn’t exist yet. The script creates it first, with the stores infetched/attached, thenv<version>.
Either way it first checks that you’re on an up-to-date
mainwith no uncommitted changes, that the stores infetched/match the manifest, and that the changelog has the version’s section, in a dry run, and then asks before creating anything. Each release gets its git tag on GitHub (git fetch --tagsto get them locally). The script is a wrapper aroundpython scripts/publish.py <data tag> [--dry-run], which you can also run directly.- Package only (the usual case): no fetched store has changed, so the release named in
Publishing
v<version>triggers the Publish workflow (.github/workflows/publish.yml), which builds the package and uploads it to PyPI. It refuses to if any fetched store can’t be downloaded. Watch it under the repository’s Actions tab.
PyPI accepts the upload through trusted publishing, so no token is needed. This is set up once on pypi.org, in the project’s Publishing settings: owner starsimhub, repository starsim_data, workflow publish.yml, environment pypi.