Skip to content

Commit f33710c

Browse files
theo-barfootpre-commit-ci[bot]ericspod
authored
Add 3D segmentation calibration tutorial (#2072)
## Description Adds an introductory tutorial for the average calibration losses proposed in Project-MONAI/MONAI#9092. The notebook explains predictive calibration, why it matters for medical image segmentation, and how to interpret reliability diagrams. It then compares a segmentation-only baseline with one validation-selected Hard L1-ACE configuration on complete 3D volumes from Medical Segmentation Decathlon Task04 Hippocampus. The example intentionally keeps the experiment simple: two training runs with identical initialization, data order, augmentations, and optimization settings. The calibrated objective uses a `1:1:0.30` Dice:CE:Hard-L1-ACE ratio. The held-out test set is not accessed until both validation-selected checkpoints have been fixed. ## Saved tutorial result | Configuration | Dice | ACE | MCE | | --- | ---: | ---: | ---: | | Segmentation only | 0.799317 | 0.117391 | 0.295517 | | Hard L1-ACE | 0.828645 | 0.090355 | 0.208534 | This corresponds to a 23% reduction in held-out ACE and a 29% reduction in MCE, while Dice increases by 0.0293 for this split and seed. The notebook explicitly presents this as a demonstration rather than a benchmark guarantee. ## Validation - Executed all 36 cells end to end on an NVIDIA RTX 5090 - Passed the repository runner with a one-epoch execution - Passed targeted notebook schema, syntax, formatting, PEP8, metadata, and output checks - Confirmed that saved outputs contain no execution errors or private local paths ## Dependency Depends on Project-MONAI/MONAI#9092 for `HardL1ACELoss` and `SoftL1ACELoss`. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Documentation** - Added a Calibration tutorial using complete 3D MRI volumes from the MSD Task04 Hippocampus dataset. - Documented calibration metrics and losses, and the comparison of a baseline with a validation-selected Dice:CE:Hard-L1-ACE configuration on separate held-out test volumes. - Added guidance for running the experiment, including a one-epoch CI smoke test, dataset requirements, recommended GPU hardware, and required MONAI APIs. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Theo Barfoot <theo.barfoot@gmail.com> Signed-off-by: Eric Kerfoot <17726042+ericspod@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Kerfoot <17726042+ericspod@users.noreply.github.com>
1 parent 312ae90 commit f33710c

3 files changed

Lines changed: 1415 additions & 0 deletions

File tree

‎README.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -101,6 +101,13 @@ Each user is responsible for checking the content of datasets and the applicable
101101
You can read details about adding a tutorial in our [CONTRIBUTING GUIDELINES](CONTRIBUTING.md).
102102

103103
### 4. List of notebooks and examples
104+
#### <ins>**Calibration**</ins>
105+
##### [Training and evaluating calibrated segmentation models](./calibration/segmentation_calibration.ipynb)
106+
This tutorial trains two models on complete 3D MRI volumes from the Medical Segmentation Decathlon
107+
`Task04_Hippocampus` dataset to demonstrate calibration metrics, reliability diagrams, the Ignite calibration
108+
handler, and L1-ACE auxiliary training. It compares a validation-selected hard L1-ACE configuration with a controlled
109+
segmentation baseline on separate held-out test volumes.
110+
104111
#### <ins>**2D classification**</ins>
105112
##### [mednist_tutorial](./2d_classification/mednist_tutorial.ipynb)
106113
This notebook shows how to easily integrate MONAI features into existing PyTorch programs.

‎calibration/README.md‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
# Segmentation model calibration
2+
3+
This folder contains a self-contained tutorial for evaluating and improving the marginal class-wise calibration of
4+
semantic segmentation models. It trains on complete 3D volumes from the Medical Segmentation Decathlon
5+
`Task04_Hippocampus` MRI dataset. The notebook downloads the approximately 28 MB archive automatically and reuses
6+
the directory configured by `MONAI_DATA_DIRECTORY`.
7+
8+
The notebook demonstrates MONAI's calibration metrics, low-level bin statistics, Ignite handler, and hard- and
9+
soft-binned L1 Average Calibration Error losses. It uses complete-volume train/validation/test cohorts and compares
10+
a segmentation baseline with one hard L1-ACE configuration chosen in validation-only preliminary experiments for
11+
its calibration improvement with minimal Dice reduction. The focused comparison uses a validation-selected
12+
Dice:CE:Hard-L1-ACE objective with an approximate ratio of 1:1:0.30. It also discusses the associated publication's
13+
1:1:1 objective, finite-bin estimates, and the limitations of auxiliary calibration training.
14+
15+
For a one-epoch CI smoke test, run:
16+
17+
```bash
18+
export MONAI_DATA_DIRECTORY=/path/to/persistent/monai-data
19+
./runner.sh -t calibration/segmentation_calibration.ipynb
20+
```
21+
22+
`runner.sh` rewrites `max_epochs` and `val_interval` to one. To reproduce the saved full experiment, open the notebook
23+
in Jupyter and run all cells without that rewrite. A CUDA GPU is strongly recommended for the two 3D training runs.
24+
25+
The notebook requires a MONAI build containing `HardL1ACELoss` and `SoftL1ACELoss`. Until those APIs are available in
26+
an official MONAI package, run it in an environment with the corresponding MONAI core contribution installed editable.

‎calibration/segmentation_calibration.ipynb‎

Lines changed: 1382 additions & 0 deletions
Large diffs are not rendered by default.

0 commit comments

Comments
 (0)