How to evaluate forecast prediction intervals with the winkler score

🎓 This tutorial aims to introduce the Winkler Score (also called the interval score) for evaluating forecast prediction intervals. By following the tutorial you will:

• ✅ Gained practical experience of using the winkler score to evaluate 1-step and h-step forecasts
• ✅ Understood how to implement the winkler score method in Python
• 🎁 Bonus: Gained expertise of using the winkler score to evaluate ARIMA prediction intervals.

License

The materials have been made available under an MIT license. The materials are as-is with no liability for the author. Please provide credit if you reuse the code in your own work.

Citation

If you reuse any of the code, or the tutorial helps you work, please provide a citation.

@software{TheOpenScienceNerd_WinklerScore
author = {Monks, Thomas},
license = {MIT},
title = {{TheOpenScienceNerd - Winkler Score Tutorial}},
url = {https://github.com/TheOpenScienceNerd/winkler_score}
}

Installation instructions

Installing dependencies

All dependencies can be found in binder/environment.yml and are pulled from conda-forge. To run the code locally, we recommend installing miniforge;

miniforge is Free and Open Source Software (FOSS) alternative to Anaconda and miniconda that uses conda-forge as the default channel for packages. It installs both conda and mamba (a drop in replacement for conda) package managers. We recommend mamba for faster resolving of dependencies and installation of packages.

navigating your terminal (or cmd prompt) to the directory containing the repo and issuing the following command:

mamba env create -f binder/environment.yml

Activate the mamba environment using the following command:

mamba activate winkler

Run Jupyter-lab

jupyter-lab

Repo overview

.
├── binder
│   └── environment.yml
├── CHANGELOG.md
├── CITATION.cff
├── LICENSE
├── 01_winkler_score.ipynb
├── ts_utility.py
└── README.md

• binder/environment.yml - contains the conda environment if you wish to work the models.
• 01_winkler_score.ipynb - tutorial notebook. Start here.
• ts_utility.py - moduel containing functions to support tutorial notebook.
• CHANGES.md - changelog with record of notable changes to project between versions.
• CITATION.cff - citation information for the code.
• LICENSE - details of the MIT permissive license of this work.