The Multiple Snow Assimilation System (MuSA), is a flexible data assimilation toolbox for experimental and operational snowpack reanalysis development. MuSA was designed to fuse gridded observations with an ensemble of simulations generated by the Flexible Snow Model (FSM2) by using different Bayesian based data assimilation algorithms.
In its current version, it also offers support for the Snow17 model and a simple temperature index model. Potentially other numerical models could be implemented (not necessarily limited to snow models).
Figure 1: Comparison between open loop and updated simulation after assimilating UAV snow depth retrievals at 5m resolution
The inputs of MuSA (using FSM2) are composed by meteorological forcing and observations to be assimilated. Both the forcing and observations must share the same geometry, with the same resolution and number of cells in the latitudinal and longitudinal axes, and should be provided in the netCDF format. Optionally it is possible to provide a mask with the same geometry of the mandatory input files to avoid to run MuSA over certain cells of your domain. The meteorological forcing needed for running MuSA is composed by:
- Incident shortwave radiation (W m-2)
- Incident longwave radiation ( W m-2)
- Precipitation (Kg-2 m-2 s-1)
- Temperature (K)
- Relative Humidity (%)
- Wind speed (m s-1)
- Atmospheric pressure (Pa)
In its current version MuSA provides support for assimilating different variables. Note that it is possible to provide more than one of the following variables at the same time, i.e. MuSA has support for joint assimilation experiments. In its current version, MuSA is able to assimilate:
- SWE (mm)
- Snow depth (m)
- Land/Snow surface temperature (K)
- Fractional snow cover area (-)
- Albedo (-)
- Sensible heat flux to the atmosphere (W m-2)
- Latent heat flux to the atmosphere (W m-2)
The support of other variables like liquid water content, density, ice content etc. is easy to implement.
There are different data assimilation and resampling algorithms implemented in MuSA. Some testing should be done when developing data assimilation experiments, as the performance may be different depending on the problem to solve, and regarding the literature there is not a clear winner. Also, the computational cost will be different, and may be a strong conditioner in some situations.
Filters:
- Particle Filter (PF)
- Ensemble Kalman filter (EnKF)
- Iterative ensemble Kalman filter (IEnKF)
Smoothers:
- Particle Batch Smoother (PBS)
- Adaptive Particle Batch Smoother (AdaPBS)
- Ensemble Smoother (ES)
- Iterative Ensemble Smoother (IES)
- Particle-adjusted Iterative Ensemble Smoother (PIES)
- Robust Adaptive Metropolis initialised by IES (IES-MCMC)
Resampling (for particle filters only):
- Bootstrapping
- Residual resampling
- Stratified resampling
- Systematic resampling
- Redraw from a normal approximation of the posterior
The outputs of MuSA are pickles compressed with blosc (the latid_lonid.pkl.bl format). Each file stores a dictionary containing the following dataframes:
- DA_Results: Contains information about the observed variables and posterior parameters (in the normal space).
- OL_Sim: Contains the reference simulation (snow simulation without any assimilation).
- mean_Prior: Contains the prior mean of the ensemble (no assimilation)
- std_Prior: Contains the prior standar deviation of the ensemble (no assimilation)
- mean_Post: Contains the updated simulation after the assimilation, i.e. weighted mean of the ensemble of simulations.
- std_Post: Contains the weighted standard deviation of the ensemble after the assimilation.
- mcmc_stat: Contains the posterior mean and standard deviation from the MCMC step (only for IES-MCMC and IES-MCMC_AI).
You can open these files by your own means, or by using the following function:
from modules.internal_fns.py import io_read
my_cell = io_read(./RESULTS/latid_lonid.pkl.blp)
lonid and latid are the longitude and latitude ids of each cell of the simulation.
Additionally it is possible to store the ensembles generated for each cell. This is an optional feature as it may be a bit memory consuming. However It may be useful in some circumstances specially for advanced users. The ensembles will be stored in the same format, and will containing Ensemble objects.
MuSA runs natively on GNU/Linux, macOS and Windows. MuSA relies on python3 with the usual scientific libraries (numpy, pandas, scipy...) and netCDF4 installed. You will also need to have gfortran in the path (or installed in the active conda environment). The easiest way to do this is to generate a dedicated conda environment.
On GNU/Linux and macOS you can use the MuSAenv.yml file of the repository to create the conda environment:
conda env create --name MuSAenv --file=MuSAenv.yml
On Windows use MuSAenv_windows.yml, which contains native Windows builds of all the dependencies (including gfortran):
conda env create --name MuSAenv --file=MuSAenv_windows.yml
Then for running MuSA simply, on every platform:
conda activate MuSAenv
python main.py
On Windows MuSA compiles FSM2 natively with gfortran. WSL, Git Bash, make or a Unix shell are not required. The executable, the gfortran runtime DLLs and the spawn multiprocessing start method are handled automatically, so python main.py works exactly as on Linux. Activating the environment is still recommended, but MuSA also works when the environment Python is called directly.
MPI is optional. If MPI is activated in the config.py file:
conda activate MuSAenv
mpiexec -n nprocess python -m mpi4py.futures main.py
Where nprocess is the number of threads to be deployed. MPI requires a working MPI implementation and mpi4py; the default multiprocessing parallelization needs neither and is fully supported on Windows.
These commands should run the reproducible example included in the repository. This example contains all the information needed by MuSA. It is composed of a few cells containing meteorological forcing and drone SfM derived snowdepth information. To change the configuration of MuSA, you should modify the config.py file. Also it is possible to modify the way MuSA generates the ensemble by modifying the constants.py file. To run a simulation without data assimilation, use implementation = "distributed" (or "point_scale") together with da_algorithm = "deterministic_OL".
An example script is also provided to run MuSA in distributed supercomputing facilities using PBS (Portable Batch System, not Particle Batch Smoother 😉) or Slurm arrays. The use of PBS/Slurm arrays is compatible with both MPI and multiprocessing. The use of PBS/Slurm arrays is compatible with both MPI and multiprocessing, allowing many different parallelization schemes depending on the need. PBS/Slurm job arrays are only available on HPC clusters and are not used on a regular Windows workstation; use parallelization = "multiprocessing" there.
If the spatial propagation is activated, it is necessary to launch the pre-processor that activates the spatial prior sampling before launching the main program.
conda activate MuSAenv
python pre_main_spatial.py
python main.py
Before publishing a new version, run the release test suite from the repository root:
conda activate MuSAenv
python run_tests.py # standard release gate
python run_tests.py --full # also run slow/optional configurations
The runner creates isolated temporary workspaces, so it does not modify
DATA/, config.py or the compiled FSM2 executable. It checks the Python
environment, compiles every Python file, validates the example data, exercises
all data assimilation algorithms and model backends, and validates the
resulting pickles. Use python run_tests.py --list to see the cases,
--quick for a fast sanity check, and --allow-known to ignore documented
known issues while still reporting them. A non-zero exit status means that at
least one unexpected test failed.
This software is licensed under the GNU General Public License v3.0 (GPL-3.0).
In addition to the terms of the GPL-3.0, we kindly request that any scientific publication, presentation, or derived work using this software includes the following citation:
Alonso-González, E., Aalstad, K., Baba, M. W., Revuelto, J., López-Moreno, J. I., Fiddes, J., Essery, R., and Gascoin, S.: The Multiple Snow Data Assimilation System (MuSA v1.0), Geosci. Model Dev., 15, 9127–9155, https://doi.org/10.5194/gmd-15-9127-2022, 2022.
This citation requirement does not alter or restrict the rights granted under the GPL-3.0 license.