TU Cookbook Python Template - TU Cookbook Development Guide

Here we provide a detailed description of the tools considered in the TU Cookbook Python template. We will highlight the files and directories relevant to each of the tools with ‘✏️’ and ‘👉’, respectively.

Code clarity¶

At the heart of the cookbook project are the Jupyter Notebooks, which can be added to notebooks directory. One can add superfluos code, like custom functions, as python scripts to the sources files under tucookbooks. Together with the pyproject.toml file this ensure that these custom function can be imported just like you do for third party Python packages. It is suggested to use type hinting and """docstrings""" for custom functions.

import seedir as sd
from pathlib import Path
from functools import partial
import re

def _highlight(item, dirname=None, filename=None, ext=None):
    outdict = {}
    basename = Path(item).stem
    fullname = Path(item).name
    extension = Path(item).suffix

    if dirname is not None:
      if re.search(r'|'.join(dirname), basename):
        outdict['folderstart'] = '👉'
    if filename is not None:
      if re.search(r'|'.join(filename), fullname):
       outdict['filestart'] = '✏️'
    if ext is not None:
      if re.search(r'|'.join(ext), extension):
        outdict['filestart'] = '✏️'
    return outdict


highlight = partial(_highlight, dirname=["src", "notebooks", "tucookbook"], filename=["pyproject.toml"], ext=["ipynb", "py"])

sd.seedir("tu-cookbook-python", style='emoji', exclude_folders=['.git'], formatter=highlight)

📁 tu-cookbook-python/
├─📄 Dockerfile
├─👉notebooks/
│ ├─✏️notebook-template.ipynb
│ └─📄 how-to-cite.md
├─📄 .gitignore
├─📄 .git
├─📄 uv.lock
├─📄 docker-compose.yml
├─📄 CITATION.cff
├─📄 CONTRIBUTING.md
├─👉src/
│ └─👉tucookbook/
│   └─✏️__init__.py
├─📄 .pre-commit-config.yaml
├─📄 CHANGELOG
├─📄 .gitlab-ci.yml
├─📄 README.md
├─✏️pyproject.toml
├─📄 COPYRIGHT.md
├─📄 environment.yml
├─📄 LICENSE
└─📄 myst.yml

Code quality¶

To ensure the best code quality we adhere to the official Python style guide PEP 8. We enforce this by using ruff, where setting can be modified in the pyproject.toml file. We encourage authors to use the pre commit hooks (.pre-commit-config.yaml). These hooks check for code quality and try to fix them whenever possible.

highlight = partial(_highlight, filename=["pyproject.toml", ".pre-commit-config.yaml"])

sd.seedir("tu-cookbook-python", style='emoji', exclude_folders=['.git'], formatter=highlight)

📁 tu-cookbook-python/
├─📄 Dockerfile
├─📁 notebooks/
│ ├─📄 notebook-template.ipynb
│ └─📄 how-to-cite.md
├─📄 .gitignore
├─📄 .git
├─📄 uv.lock
├─📄 docker-compose.yml
├─📄 CITATION.cff
├─📄 CONTRIBUTING.md
├─📁 src/
│ └─📁 tucookbook/
│   └─📄 __init__.py
├─✏️.pre-commit-config.yaml
├─📄 CHANGELOG
├─📄 .gitlab-ci.yml
├─📄 README.md
├─✏️pyproject.toml
├─📄 COPYRIGHT.md
├─📄 environment.yml
├─📄 LICENSE
└─📄 myst.yml

Dependencies¶

We can manage dependencies with either conda or uv by modifying the environment.yml and the pyproject.toml file. Pinning of package version should be limited to only a minimum version requirement. To make a snapshot of the current package versions one can use uv and create a uv.lock file.

We use Docker for complex system requirements and for checking JupyterHub compatibility. See the previous chapter on how to quickly launch a JupyterLab with Docker compose.

highlight = partial(_highlight, filename=["Dockerfile", "docker-compose.yml", "environment.yml", "pyproject.toml", "uv.lock"])

sd.seedir("tu-cookbook-python", style='emoji', exclude_folders=['.git'], formatter=highlight)

📁 tu-cookbook-python/
├─✏️Dockerfile
├─📁 notebooks/
│ ├─📄 notebook-template.ipynb
│ └─📄 how-to-cite.md
├─📄 .gitignore
├─📄 .git
├─✏️uv.lock
├─✏️docker-compose.yml
├─📄 CITATION.cff
├─📄 CONTRIBUTING.md
├─📁 src/
│ └─📁 tucookbook/
│   └─📄 __init__.py
├─📄 .pre-commit-config.yaml
├─📄 CHANGELOG
├─📄 .gitlab-ci.yml
├─📄 README.md
├─✏️pyproject.toml
├─📄 COPYRIGHT.md
├─✏️environment.yml
├─📄 LICENSE
└─📄 myst.yml

Code testing¶

We can check the executability of the notebooks locally by running

myst start --execute

In addition, we have a GitLab CI that runs the same code on every push to main, merge request, and by a scheduled trigger every night.

highlight = partial(_highlight, filename=["myst.yml", ".gitlab-ci.yml"])

sd.seedir("tu-cookbook-python", style='emoji', exclude_folders=['.git'], formatter=highlight)

📁 tu-cookbook-python/
├─📄 Dockerfile
├─📁 notebooks/
│ ├─📄 notebook-template.ipynb
│ └─📄 how-to-cite.md
├─📄 .gitignore
├─📄 .git
├─📄 uv.lock
├─📄 docker-compose.yml
├─📄 CITATION.cff
├─📄 CONTRIBUTING.md
├─📁 src/
│ └─📁 tucookbook/
│   └─📄 __init__.py
├─📄 .pre-commit-config.yaml
├─📄 CHANGELOG
├─✏️.gitlab-ci.yml
├─📄 README.md
├─📄 pyproject.toml
├─📄 COPYRIGHT.md
├─📄 environment.yml
├─📄 LICENSE
└─✏️myst.yml

Attribution¶

We can license 2 things: code by adding a LICENSE (optionally also COPYRIGHT.md) and the content which uses the myst.yml (see the following guide on the myst.yml).

highlight = partial(_highlight, filename=["LICENSE", "COPYRIGHT.md", "myst.yml"])

sd.seedir("tu-cookbook-python", style='emoji', exclude_folders=['.git'], formatter=highlight)

📁 tu-cookbook-python/
├─📄 Dockerfile
├─📁 notebooks/
│ ├─📄 notebook-template.ipynb
│ └─📄 how-to-cite.md
├─📄 .gitignore
├─📄 .git
├─📄 uv.lock
├─📄 docker-compose.yml
├─📄 CITATION.cff
├─📄 CONTRIBUTING.md
├─📁 src/
│ └─📁 tucookbook/
│   └─📄 __init__.py
├─📄 .pre-commit-config.yaml
├─📄 CHANGELOG
├─📄 .gitlab-ci.yml
├─📄 README.md
├─📄 pyproject.toml
├─✏️COPYRIGHT.md
├─📄 environment.yml
├─✏️LICENSE
└─✏️myst.yml

Citing¶

To facilitate minting a Zenodo DOI we can add metadata to the CITATION.cff. To be developed. The reader of the cookbook should also be informed on how to cite he book properly by adapting the how-to-cite.md in the notebooks directory.

highlight = partial(_highlight, filename=["CITATION.cff", "how-to-cite.md"])

sd.seedir("tu-cookbook-python", style='emoji', exclude_folders=['.git'], formatter=highlight)

📁 tu-cookbook-python/
├─📄 Dockerfile
├─📁 notebooks/
│ ├─📄 notebook-template.ipynb
│ └─✏️how-to-cite.md
├─📄 .gitignore
├─📄 .git
├─📄 uv.lock
├─📄 docker-compose.yml
├─✏️CITATION.cff
├─📄 CONTRIBUTING.md
├─📁 src/
│ └─📁 tucookbook/
│   └─📄 __init__.py
├─📄 .pre-commit-config.yaml
├─📄 CHANGELOG
├─📄 .gitlab-ci.yml
├─📄 README.md
├─📄 pyproject.toml
├─📄 COPYRIGHT.md
├─📄 environment.yml
├─📄 LICENSE
└─📄 myst.yml

Reviewing¶

In the future we will propose a GitLab commit standard and branching model. For now try to be concise with commit messages on the purpose of the changes as well as make merge requests that can be reviewed by others. We already highlighted some primers in the CONTRIBUTING.md of the template to guide contributors.

Prevent pushing large files (50 Mb) this includes executed notebooks. The pre-commit hooks prevent accidental commiting of executed notebooks. The latter can be detrimental for the GitLab project as it can significantly inflate the repository size.

highlight = partial(_highlight, filename=[".pre-commit-config.yaml", "CONTRIBUTING.md", "CHANGELOG"])

sd.seedir("tu-cookbook-python", style='emoji', exclude_folders=['.git'], formatter=highlight)

📁 tu-cookbook-python/
├─📄 Dockerfile
├─📁 notebooks/
│ ├─📄 notebook-template.ipynb
│ └─📄 how-to-cite.md
├─📄 .gitignore
├─📄 .git
├─📄 uv.lock
├─📄 docker-compose.yml
├─📄 CITATION.cff
├─✏️CONTRIBUTING.md
├─📁 src/
│ └─📁 tucookbook/
│   └─📄 __init__.py
├─✏️.pre-commit-config.yaml
├─✏️CHANGELOG
├─📄 .gitlab-ci.yml
├─📄 README.md
├─📄 pyproject.toml
├─📄 COPYRIGHT.md
├─📄 environment.yml
├─📄 LICENSE
└─📄 myst.yml

Packaging¶

We automatically create an image for the TUWEL JupyterHub with the GitLab CI/CD upon every release (using semantic versioning). For this to work you need to modify the Dockerfile and pyproject.toml after your project’s needs. In the future this will be adapted to also create an image for the dataLab BinderHub for greater speed in spinning up a BinderHub JupyterLab.

highlight = partial(_highlight, filename=["Dockerfile", "pyproject.toml", ".gitlab-ci.yml"])

sd.seedir("tu-cookbook-python", style='emoji', exclude_folders=['.git'], formatter=highlight)

📁 tu-cookbook-python/
├─✏️Dockerfile
├─📁 notebooks/
│ ├─📄 notebook-template.ipynb
│ └─📄 how-to-cite.md
├─📄 .gitignore
├─📄 .git
├─📄 uv.lock
├─📄 docker-compose.yml
├─📄 CITATION.cff
├─📄 CONTRIBUTING.md
├─📁 src/
│ └─📁 tucookbook/
│   └─📄 __init__.py
├─📄 .pre-commit-config.yaml
├─📄 CHANGELOG
├─✏️.gitlab-ci.yml
├─📄 README.md
├─✏️pyproject.toml
├─📄 COPYRIGHT.md
├─📄 environment.yml
├─📄 LICENSE
└─📄 myst.yml

Publication¶

The template facilitates deployment of the myst Jupyter book to GitLab pages. One can test whether the book builds by running myst locally.

myst build --execute

highlight = partial(_highlight, filename=["myst.yml", ".gitlab-ci.yml"])

sd.seedir("tu-cookbook-python", style='emoji', exclude_folders=['.git'], formatter=highlight)

📁 tu-cookbook-python/
├─📄 Dockerfile
├─📁 notebooks/
│ ├─📄 notebook-template.ipynb
│ └─📄 how-to-cite.md
├─📄 .gitignore
├─📄 .git
├─📄 uv.lock
├─📄 docker-compose.yml
├─📄 CITATION.cff
├─📄 CONTRIBUTING.md
├─📁 src/
│ └─📁 tucookbook/
│   └─📄 __init__.py
├─📄 .pre-commit-config.yaml
├─📄 CHANGELOG
├─✏️.gitlab-ci.yml
├─📄 README.md
├─📄 pyproject.toml
├─📄 COPYRIGHT.md
├─📄 environment.yml
├─📄 LICENSE
└─✏️myst.yml