Landau Damping + FFT periodic solver crashes on Lumi

[s-mayani]

The Landau Damping test case with the FFT periodic Poisson solver crashes on Lumi since the new update in January 2026 (could run without issues in December). This only happens for certain problem sizes and node counts.

Specifically, it is happening for

• 1024^3 grid points, with 8 particles per cell, 10 timesteps
• 8, 16, 32, 64, 128 nodes (x8 to get number of GPUs)

The error is: Memory access fault by GPU node-5 (Agent handle: 0x745fc0) on address 0x147299919000. Reason: Unknown.

The reason for this is unknown, but does not seem to be reproducible with the heffte benchmarks when using the same problem size. Furthermore, the simulation still runs for a certain amount of timesteps (e.g. 6 or 8 timesteps) before crashing with this error, so initially it seems that there is not a problem. Such an error was not seen when running the same problem sizes and node counts on Lumi before the update.

To reproduce it, here the modules I load on Lumi:

module load LUMI/25.03 partition/G cpeAMD rocm buildtools/25.03

Then, clone IPPL master and create a build directory, and run cmake with the following options:

cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_STANDARD=20 -DCMAKE_CXX_COMPILER=hipcc -DBUILD_SHARED_LIBS=ON -DCMAKE_HIP_ARCHITECTURES=gfx90a -DCMAKE_HIP_FLAGS=--offload-arch=gfx90a -DKokkos_ARCH_ZEN3=ON -DKokkos_ARCH_AMD_GFX90A=ON -DKokkos_ENABLE_HIP=ON -DIPPL_PLATFORMS="HIP;OPENMP" -DIPPL_ENABLE_TESTS=ON -DIPPL_ENABLE_UNIT_TESTS=OFF -DIPPL_ENABLE_FFT=ON -DIPPL_ENABLE_SOLVERS=ON -DIPPL_ENABLE_ALPINE=ON -DHeffte_ENABLE_ROCM=ON -DHeffte_ENABLE_GPU_AWARE_MPI=ON -DIPPL_ENABLE_ALIGNED_COMMUNICATION_BUFFERS=ON -DCMAKE_EXE_LINKER_FLAGS="-L/opt/cray/pe/mpich/8.1.32/ofi/amd/6.0/lib -L/opt/cray/pe/mpich/8.1.32/gtl/lib -L/opt/cray/pe/libsci/25.03.0/AMD/6.0/x86_64/lib -L/opt/cray/pe/dsmml/0.3.0/dsmml//lib -L/opt/cray/xpmem/2.8.2-1.0_5.1__g84a27a5.shasta/lib64 -lsci_amd_mpi -lsci_amd -ldl -lmpi_amd -lmpi_gtl_hsa -ldsmml -lxpmem -L/opt/rocm-6.3.4/lib/lib -L/opt/rocm-6.3.4/lib/lib64 -L/opt/rocm-6.3.4/lib/llvm/lib"

The jobscript I am using is:

#!/bin/bash
# ---
# sbatch settings for AMD Lumi
# ---
#SBATCH --job-name=landauFFT
#SBATCH --output=landauFFT_%j.out
#SBATCH --error=landauFFT_%j.err
#SBATCH --time=00:15:00
#SBATCH --nodes 8
#SBATCH --partition=standard-g
#SBATCH --ntasks-per-core=1
#SBATCH --ntasks-per-node=8
#SBATCH --gpus-per-node=8
#SBATCH --cpus-per-task=8
#SBATCH --account=[account number]
#SBATCH --hint=nomultithread
#SBATCH --hint=exclusive

CPU_BIND="map_cpu:49,57,17,25,1,9,33,41"

export MPICH_GPU_SUPPORT_ENABLED=1

ulimit -s unlimited

mkdir -p data

# Threading and pinning
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
export OMP_PLACES=threads               # bind to full cores
export OMP_PROC_BIND=spread            # keep threads close to the rank
# (optional, redundant for OpenMP backend)
export KOKKOS_NUM_THREADS=$SLURM_CPUS_PER_TASK

cat << EOF > select_gpu
#!/bin/bash

export ROCR_VISIBLE_DEVICES=\$SLURM_LOCALID
exec \$*
EOF
chmod +x ./select_gpu

export exedir=[your_exe_dir]

srun ./select_gpu ${exedir}/LandauDamping 1024 1024 1024 8589934592 10 FFT 1.0 LeapFrog --overallocate 2.0 --info 5

where the "exedir" should correspond to the alpine folder in the build directory of IPPL, where the LandauDamping executable can be found.

[paubauer]

please change the -DCMAKE_CXX_COMPILER from hipcc to amdclang++. I only had issues with using hipcc there instead of the proper wrapper

[s-mayani]

I tried with amdclang++ but didn't solve the isssue.
Also attaching the backtrace with the debugger (for 16 nodes).

output.txt
error.txt

[paubauer]

thanks for the logs, but I don't see anything in the backtrace for the actual segmentation fault. Will try to reproduce on a different system

[s-mayani]

A smaller reproducer: when running the weak scaling I also had this crash happen with the following configuration (jobscript below).

#!/bin/bash
# ---
# sbatch settings for AMD Lumi
# ---
#SBATCH --job-name=landauFFT
#SBATCH --output=landauFFT_%j.out
#SBATCH --error=landauFFT_%j.err
#SBATCH --time=00:59:00
#SBATCH --nodes 2
#SBATCH --partition=standard-g
#SBATCH --ntasks-per-core=1
#SBATCH --ntasks-per-node=8
#SBATCH --gpus-per-node=8
#SBATCH --cpus-per-task=8
#SBATCH --account=project_465002453
#SBATCH --hint=nomultithread
#SBATCH --hint=exclusive

CPU_BIND="map_cpu:49,57,17,25,1,9,33,41"

export MPICH_GPU_SUPPORT_ENABLED=1

ulimit -s unlimited

mkdir -p data

# Threading and pinning
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
export OMP_PLACES=threads               # bind to full cores
export OMP_PROC_BIND=spread            # keep threads close to the rank
# (optional, redundant for OpenMP backend)
export KOKKOS_NUM_THREADS=$SLURM_CPUS_PER_TASK

cat << EOF > select_gpu
#!/bin/bash

export ROCR_VISIBLE_DEVICES=\$SLURM_LOCALID
exec \$*
EOF
chmod +x ./select_gpu

export exedir=[path_to_ippl/build/alpine]

module load LUMI/25.03 partition/G cpeAMD rocm buildtools/25.03

srun ./select_gpu ${exedir}/LandauDamping 256 256 512 268435456 10 FFT 1.0 LeapFrog --overallocate 2.0 --info 5

[s-mayani]

Unfortunately, using the following environment variables

HSA_SDMA=0
AMD_SERIALIZE_COPY=0

in the jobscript as suggested by @paubauer did not help, the crash is still there.

[srikrrish]

Just to update on this issue. It also happens with PIF for certain problem sizes and node counts (seems it is more prone to happen for large problem sizes and high node counts). As Sonali, mentioned it even runs for 6 time steps or so and then crashes. Here is what I see in the error.

Memory access fault by GPU node-9 (Agent handle: 0x7970f0) on address 0x14b76dc6e000. Reason: Unknown.
srun: error: nid006960: task 5: Aborted
srun: Terminating StepId=16786370.0
slurmstepd: error: *** STEP 16786370.0 ON nid006960 CANCELLED AT 2026-03-17T11:16:14 ***
srun: error: nid006960: tasks 0-4,6-7: Terminated
srun: Force Terminated StepId=16786370.0

Steps to reproduce the error. Load the following modules on LUMI

module load LUMI/25.03 partition/G cpeAMD rocm buildtools/25.03

Clone https://github.com/srikrrish/ippl/tree/nufft_native and compile using

cmake ../ippl/ \
      -DCMAKE_BUILD_TYPE=Release \
      -DCMAKE_CXX_STANDARD=20 \
      -DHeffte_ENABLE_GPU_AWARE_MPI=ON \
      -DCMAKE_CXX_COMPILER=hipcc \
      -DBUILD_SHARED_LIBS=ON \
      -DCMAKE_HIP_ARCHITECTURES=gfx90a \
      -DCMAKE_HIP_FLAGS=--offload-arch=gfx90a \
      -DKokkos_ARCH_ZEN3=ON \
      -DKokkos_ARCH_AMD_GFX90A=ON \
      -DKokkos_ENABLE_HIP=ON \
      -DIPPL_PLATFORMS="HIP" \
      -DIPPL_ENABLE_TESTS=ON \
      -DIPPL_ENABLE_FFT=ON  \
      -DIPPL_ENABLE_SOLVERS=ON \
      -DIPPL_ENABLE_ALPINE=ON \
      -DHeffte_ENABLE_ROCM=ON \
      -DKokkos_VERSION=git.4.7.01 \
      -DHeffte_VERSION=git.v2.4.1 \
      -DCMAKE_EXE_LINKER_FLAGS="-L/opt/cray/pe/mpich/8.1.32/ofi/amd/6.0/lib -L/opt/cray/pe/mpich/8.1.32/gtl/lib -L/opt/cray/pe/libsci/25.03.0/AMD/6.0/x86_64/lib -L/opt/cray/pe/dsmml/0.3.0/dsmml//lib -L/opt/cray/xpmem/2.8.2-1.0_5.1__g84a27a5.shasta/lib64 -lsci_amd_mpi -lsci_amd -ldl -lmpi_amd -lmpi_gtl_hsa -ldsmml -lxpmem -L/opt/rocm-6.3.4/lib/lib -L/opt/rocm-6.3.4/lib/lib64 -L/opt/rocm-6.3.4/lib/llvm/lib"

make -j8 LandauDampingPIF

Minimal reproducer on 1 node with the following jobscript

#!/bin/bash
#SBATCH --job-name=landau_pif
#SBATCH --error=landau_pif.error
#SBATCH --output=landau_pif.out
#SBATCH --partition=standard-g  # partition name
#SBATCH --time=00:15:00
#SBATCH --nodes 1
#SBATCH --ntasks-per-node=8     # 8 MPI ranks per node, 64 total (8x8)
#SBATCH --gpus-per-node=8       # Allocate one gpu per MPI rank per node
#SBATCH --account=project_465002453
#SBATCH --hint=nomultithread

module load LUMI/25.03 partition/G cpeAMD rocm buildtools/25.03
CPU_BIND="map_cpu:49,57,17,25,1,9,33,41"
export MPICH_GPU_SUPPORT_ENABLED=1
ulimit -s unlimited

cat << EOF > select_gpu
#!/bin/bash
export ROCR_VISIBLE_DEVICES=\$SLURM_LOCALID
exec \$*
EOF
chmod +x ./select_gpu
srun ./select_gpu ${EXE_DIR}/LandauDampingPIF 256 256 256 134217728 10 0.0245436926 B-spline 1 1e-4 --info 5
rm -rf ./select_gpu

where the EXE_DIR should point to alpine/ElectrostaticPIF where we can find the LandauDampingPIF executable. The following is the output that I see

LandauDampingPIF{0}> Landau damping
LandauDampingPIF{0}> nt 10 Np= 134217728 Fourier modes = ( 512 , 512 , 512 )
LandauDampingPIF{0}> particles created and initial conditions assigned
LandauDampingPIF{0}> After init shape function
LandauDampingPIF{0}> After init NUFFT
LandauDampingPIF{0}> After update
LandauDampingPIF{0}> After scatter
LandauDampingPIF{0}> After gather
LandauDampingPIF{0}> Starting iterations ...
LandauDampingPIF{0}> Finished time step: -2 time: 0.0245437
LandauDampingPIF{0}> Finished time step: -1 time: 0.0490874
LandauDampingPIF{0}> Finished time step: 0 time: 0.0736311
LandauDampingPIF{0}> Finished time step: 1 time: 0.0981748
LandauDampingPIF{0}> Finished time step: 2 time: 0.122718
LandauDampingPIF{0}> Finished time step: 3 time: 0.147262
LandauDampingPIF{0}> Finished time step: 4 time: 0.171806
LandauDampingPIF{0}> Finished time step: 5 time: 0.19635
LandauDampingPIF{0}> Finished time step: 6 time: 0.220893

and the error is the one which is posted above.

[paubauer]

@srikrrish I ran your example under rocgdb, and there it finished correctly. Running without rocgdb crashed for me as well on step 6. So we have an uninitialized variable somewhere that the debugger initializes correctly for us. Do you know if clang static analyzer works with the code?

[paubauer]

It seems to be stochastic how often it crashes. I got it crash now as well in the debugger, and it also seems to be dependent on whether certain environment variables are set