ChangeSAM: Adapting the Segment Anything Model to Street Scene Image Change Detection

International Conference on Intelligent Transportation Systems 2024 Paper

ChangeSAM adapts the powerful Segment Anything Model (SAM) for the specialized task of street scene image change detection. A light-weight fusion layer is used to fuse the bi-temporal information. Prompts are replaced with a learnt "change prompt". Low-Rank Adaptation (LoRA) is used to adapt the image encoder. The heavy SAM image encoder is replaced with MobileSAM to enable ChangeSAM for use on mobile platforms such as autonomous cars. The figures below show the two variants of ChangeSAM.

ChangeSAM Pre Decoder Fusion (PreDF)

ChangeSAM Post Decoder Fusion (PostDF)

Only purple modules are adapted during training.

The accompanying paper presents various experiments and results.

---

Table of Contents

• Quick Start
• Installation
• Dataset
• Model Initialization
• Tools
• License
• Citing ChangeSAM
• Acknowledgements

---

Quick Start

Train ChangeSAM PostDF

# download VL-CMU-CD to ./VL-CMU-CD
gdown --id 1iQa9OAHNAhzO0tCufpgpowkHvkeUNxVG
unzip VL-CMU-CD.zip
# download mobile-sam checkpoint to ./mobile_sam.pt
wget https://github.com/ChaoningZhang/MobileSAM/raw/refs/heads/master/weights/mobile_sam.pt
pip install git+https://github.com/tritolol/ChangeSAM
changesam_train.py VL-CMU-CD mobile_sam.pt --decoder postdf

Test ChangeSAM

# expects checkpoint ./best_adapted_checkpoint.pt (created by changesam_train.py)
changesam_test.py VL-CMU-CD mobile_sam.pt --decoder postdf

Installation

ChangeSAM requires:

• Python: version >=3.10.8
• PyTorch: version >=2.0.1
• TorchVision: version >=0.15.2

Note: For optimal performance, install PyTorch and TorchVision with CUDA support. Follow the official PyTorch installation instructions to set up your environment.

Installing ChangeSAM

You can install ChangeSAM directly from GitHub:

pip install git+https://github.com/tritolol/ChangeSAM

Or, to install locally:

git clone [email protected]:tritolol/ChangeSAM.git
cd changesam
pip install -e .

Dataset

ChangeSAM is trained and evaluated on the VL-CMU-CD dataset which can be downloaded from here.

Model Initialization

ChangeSAM can be initialized either from the MobileSAM checkpoint (recommended, download here) or the original SAM checkpoint from the SAM repository (untested). During training, ChangeSAM checkpoints containing weights for the tuned prompt tokens, the fusion layer and optionally the LoRA weights of certain layers in the image encoder are stored and can be used for testing.

Tools

The repository provides several command-line tools located in the tools/ directory. These scripts are installed to your PATH during setup and can be configured via command-line arguments (use -h for help).

• Training:

  changesam_train.py

  Description: Train the ChangeSAM model using the provided dataset.

• Testing:

  changesam_test.py

  Description: Evaluate a trained model.

• Embedding Precomputation:

  changesam_precompute_embeddings.py

  Description: Precompute image embeddings using the SAM image encoder across one or multiple supported pytorch devices. These embeddings are not required but can speed up training significantly if neither augmentation nor image encoder adaptation is desired.

Usage example to see all available options for a tool:

changesam_train.py -h

License

The repository is licensed under the Apache 2.0 license.

Citing ChangeSAM

@INPROCEEDINGS{10920237,
  author={Bauer, Adrian and Krabbe, Jan-Christoph and Kummert, Anton},
  booktitle={2024 IEEE 27th International Conference on Intelligent Transportation Systems (ITSC)}, 
  title={ChangeSAM: Adapting the Segment Anything Model to Street Scene Image Change Detection}, 
  year={2024},
  volume={},
  number={},
  pages={1672-1677},
  keywords={Adaptation models;Image segmentation;Accuracy;Foundation models;Computational modeling;Predictive models;Computational efficiency;Decoding;Vehicle dynamics;Tuning;Foundation Model Adaptation;Change Detection;ChangeSAM;SAM;Segment Anything Model},
  doi={10.1109/ITSC58415.2024.10920237}}

Acknowledgements

SAM (Segment Anything) [bib]

@article{kirillov2023segany,
title={Segment Anything}, 
author={Kirillov, Alexander and Mintun, Eric and Ravi, Nikhila and Mao, Hanzi and Rolland, Chloe and Gustafson, Laura and Xiao, Tete and Whitehead, Spencer and Berg, Alexander C. and Lo, Wan-Yen and Doll{\'a}r, Piotr and Girshick, Ross},
journal={arXiv:2304.02643},
year={2023}
}

MobileSAM [bib]

@article{mobile_sam,
title={Faster Segment Anything: Towards Lightweight SAM for Mobile Applications},
author={Zhang, Chaoning and Han, Dongshen and Qiao, Yu and Kim, Jung Uk and Bae, Sung-Ho and Lee, Seungkyu and Hong, Choong Seon},
journal={arXiv preprint arXiv:2306.14289},
year={2023}
}

LoRA [bib]

@misc{hu2021loralowrankadaptationlarge,
 title={LoRA: Low-Rank Adaptation of Large Language Models}, 
 author={Edward J. Hu and Yelong Shen and Phillip Wallis and Zeyuan Allen-Zhu and Yuanzhi Li and Shean Wang and Lu Wang and Weizhu Chen},
 year={2021},
 eprint={2106.09685},
 archivePrefix={arXiv},
 primaryClass={cs.CL},
 url={https://arxiv.org/abs/2106.09685}, 
 }