LoTUS: Large-Scale Machine Unlearning with a Taste of Uncertainty

Poster and more can be found in the [project page](https://cspartalis.github.io/publications/).

🍊 In Homer's Odyssey, the lotus fruit induces forgetfulness, erasing the memory of Ulysses' comrades and their desire to return home.

🤖 In Machine Learning, forgetting is not always a bug; it can be a feature. So, why and how ML models need to forget?

🕵️‍♀️ User Data Removal upholds the right to be forgotten. When a user opts out of data collection, their information must not only be deleted from databases but also eliminated from any ML models trained on it.

📝 TL;DR

🎯 LoTUS is a novel Machine Unlearning method designed to eliminate the influence of specific training samples from pre-trained models.

💡 Addressing Memorization: Deep Neural Networks typically memorize training data, leading to over-confident predictions.

✅ Uncertainty Introduction: LoTUS mitigates over-confidence by smoothing the output probability distributions of the samples to be unlearned.

💡 Selective Information Removal: Only the unique, sample-specific information in the data to be unlearned should be removed. Shared information, common across other training samples, should be retained to preserve model utility.

✅ Information-Theoretic Bound: LoTUS introduces uncertainty up to a well-defined information-theoretic bound, ensuring effective unlearning while preserving generalization.

🚀 Key Contributions

🌱 Novel Method: Logits Tempering Unlearning Strategy (LoTUS), a novel entropy-based unlearning method with theoretical guarantees.

🌱 Novel Metric: Retrain-Free Jensen Shannon Divergence (RF-JSD), designed for real-world scenarios where retraining a model from scratch without the forget samples is impractical or infeasible. Existing metrics often assume access to a retrained reference model.

🌱 Novel Benchmark: Large-scale evaluation on ImageNet1k with limited data access, simulating real-world unlearning scenarios.

🏆 Why LoTUS?

🥇 Effectiveness | 🥇 Efficiency | 🥇 Scalability

🔬 Experimental Setup

⚠️ Attention!: LoTUS incorporates an accuracy threshold. Model's accuracy is in [0, 1]. If accuracy is computed in [0, 100] in your codebase, then you need to uncomment line 293 in lotus_class.py.

Unlearning Methods

All the methods are implemented in src/unlearning_methods/.

Method	File	Paper	Code
LoTUS	our_class.py	CVPR, arXiv	💻
Finetuning	naive_unlearning_class.py	📕	💻
NegGrad+	naive_unlearning_class.py	📕	💻
Rnd Labeling	naive_unlearning_class.py	📕	💻
Bad Teacher	bad_teaching_class.py	📕	💻
SCRUB	scrub_class.py	📕	💻
SSD	ssd_class.py	📕	💻
UNSIR	unsir_class.py	📕	💻
SalUn	salun_class.py	📕	💻

Evaluation Metrics

All the evaluation metrics are implemented in src/helpers/eval.py.

Metric	Paper	Code
Retrain-Free Jensen-Shannon Divergence (RF-JSD)	CVPR, arXiv	💻 log_js_proxy()
Jensen-Shannon Divergence (JSD)	📕	💻
MIA	📕 📕	💻 💻

🛠️ Reproducibility

All results presented in the paper can be reproduced. They have been documented in Jupyter notebooks.

Below is a mapping of the Tables and Figures from the paper to the corresponding notebooks:

Tables and Figures	Corresponding Notebooks
Table 1: Performance Summary	notebooks/results.ipynb
Table 2: LoTUS with Synthetic Data	notebooks/results.ipynb
Table 3: Scaling up the Forget set	notebooks/results.ipynb
Table 4: Large-Scale Unlearning	notebooks/results_imagenet.ipynb
Table 4: Accuracy Metrics	notebooks/results.ipynb
Table 6: Scaling up the Forget set (Appendix)	notebooks/results.ipynb
Figure 3: Duplicates in MUFAC	notebooks/clean_MUFAC.ipynb
Figure 4: MUFAC Class Distribution	notebooks/EDA_MUFAC.ipynb
Figure 5: Failure Analysis	notebooks/check_orthogonality.ipynb
Figure 7: Streisand Effect	notebooks/streisand.ipynb

The results are reproduced following the steps 1--4:

Step 1: Environment Setup

The environment 'MaUn' contains Conda and Pip packages. You can install them using the environment.yml file:

1. In the last line of the file, replace <username> with your username.
2. conda env create -f environment.yml
3. conda activate MaUn If this fails, you can install the Conda and Pip packages explicitly using the conda_requirements.txt and pip_requirements.txt explicitly. If the installation of a package fails, then try to remove the version (e.g., mlflow==2.8.0 → mlflow) in the corresponding file.

We use Python 3.11.5 and CUDA 12.1 in Ubuntu 24.04.1 LTS.

Step 2: Datasets

• CIFAR-10/100
  1. These datasets are downloaded automatically in ~/home/data/.
  2. You can define how many training samples are designated for unlearning using the frac_per_class_forget variable of the UnlearningDataLoader class in src/helpers/data_utils.py.
• MUFAC:
  1. Download the original dataset following the instruction here.
  2. Move the custom_korean_family_dataset_resolution_128/ folder into the ~/home/data/ folder.
  3. Run notebooks/clean_MUFAC.ipynb to clean the dataset and create the ~/home/data/custom_korean_family_dataset_resolution_128_clean\ folder.
• ImageNet1k:
  1. Download the dataset from image-net.org.
  2. Our ~/home/data/pytorch_imagenet1k folder looks like this:

  pytorch_imagenet1k/
  ├── ILSVRC2012_devkit_t12.tar.gz
  ├── ILSVRC2012_img_train.tar.gz
  ├── ILSVRC2012_img_val.tar.gz
  ├── train/
  ├── val/
  └── meta.bin
  

Step 3: MLflow Setup

Set your mlflow_tracking_uri in src/helpers/mlflow_utils. You assign it once, only in this file.

Step 4: Running the Project

Run the src/bash_scripts/reproduce_results.sh file.

Hardware

For ImageNet1k781 experiments, we used an NVIDIA RTX A6000 48GB GPU. The remaining experiments were performed on an NVIDIA RTX 4080 16GB GPU. We also used an Intel i7-12700K CPU and 32GB RAM.

‼️ Citation

If you use our work, please cite:

@inproceedings{spartalis2025lotus,
  title={LoTUS: Large-Scale Machine Unlearning with a Taste of Uncertainty},
  author={Spartalis, Christoforos N and Semertzidis, Theodoros and Gavves, Efstratios and Daras, Petros},
  booktitle={Proceedings of the Computer Vision and Pattern Recognition Conference},
  pages={10046--10055},
  year={2025}
}