mpi: fix local_size_many_transposed when yblock == n[1] (#174)#413
Open
meng004 wants to merge 1 commit into
Open
mpi: fix local_size_many_transposed when yblock == n[1] (#174)#413meng004 wants to merge 1 commit into
meng004 wants to merge 1 commit into
Conversation
The transposed output distribution was gated by a strict yblock < n[1], so passing yblock == n[1] fell through to the non-transposed fallback and left dims[1].ob = n[1]. That made rank 1 non-idle and report local_ny = n1 instead of 0, disagreeing with fftw_mpi_plan_many_transpose which distributes n1 by yblock unconditionally. Use <= so the boundary case takes the transposed distribution the planner already produces.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The transposed output distribution was gated by a strict yblock < n[1], so passing yblock == n[1] fell through to the non-transposed fallback and left dims[1].ob = n[1]. That made rank 1 non-idle and report local_ny = n1 instead of 0, disagreeing with fftw_mpi_plan_many_transpose which distributes n1 by yblock unconditionally. Use <= so the boundary case takes the transposed distribution the planner already produces.
Problem
fftwf_mpi_local_size_many_transposedreturns an inconsistent partition of thetransposed (y) dimension when the explicitly-specified
yblockequalsn[1]. Asreported in #174, for grid
n = {6, 5, 4}withblock0 = block1 = 5on 2 MPIranks, the transposed y-domain local size is reported as 5 on both ranks
(sum = 10 ≠
n1= 5). Rank 1's y-size should be 0. The reporter also observesthat swapping the dimension order (so
n1lands on the input side instead ofthe output side) produces the correct result — pinpointing the output-dimension
sizing path.
Root cause
mpi/api.c,XM(local_size_many_transposed)(local_size_many_transposed):simple_dims()initializesdims[i].ob = n[i]. When the caller passesyblock == n[1](5), the guard5 < 5is false, so theelsebranch runs,dims[1].obis never set toyblockand remainsn[1]— i.e. "dimension 1 notdistributed, full extent on every non-idle rank". The subsequent
local_size_guruthen keeps dim 0 distributed on output while treating dim 1 asfull, so rank 1 is non-idle and reports
local_ny = 5instead of 0.This is a genuine defect, not a documented boundary behavior, because the sizing
function disagrees with the actual transpose planner for the same arguments:
fftwf_mpi_plan_many_transpose(n0, n1, howmany, block0, block1, …)distributesn1byyblockunconditionally (num_blocks(5, 5) = 1block ⇒ rank 0 gets all5 y-rows, rank 1 gets 0). A user who sizes allocations from
local_sizeandthen plans gets a mismatch. The reporter's dims-swap observation maps exactly to
this guard: the input-side assignment
dims[0].ib = xblockis unconditional andyields the correct size-0 idle rank when
block == n, while the output-sidedim-1 assignment is gated by the strict
<— son1is handled correctly onthe input side and incorrectly on the output side.
Fix
Change the strict
<to<=soyblock == n[1]also takes the transposeddistribution (
dims[1].ob = yblock), matching what the planner does:Minimal and safe by construction:
FFTW_MPI_DEFAULT_BLOCK(yblock = 0), both0 < n1and0 <= n1are true — same branch,default_sz()fills the blockas before.
yblock < n1path unchanged — both old and new take theifbranch.yblock == n1changes: from a silent non-transposedfallback (wrong sizing) to the correct 1-block transposed distribution that
the planner already uses.
Test evidence
Validated on FFTW 3.3.10 (single precision + MPI,
--enable-mpi --enable-float,MPICH,
mpiexec -n 2) with two arms: pristine vs. patched.n1=5:sum(local_ny)disjointlocal_size_many(n0=6){5,5,4}, default blockslocal_size_many_transposed, planmany_transpose, execute, compare to analytic transposeBoth controls are byte-identical across arms (verified by
diff), so the changeraises no false alarm on the non-transposed path or on a symmetric grid. The
round-trip on the fixed arm proves the sizes are usable end-to-end; the
round-trip on the buggy arm proves the defect is harmful (a correct consumer
reads data the planner never produced), not a benign over-estimate.
How to reproduce
Build with
mpicc repro.c -lfftw3f_mpi -lfftw3fand runmpiexec -n 2 ./a.out.Before the fix both ranks report
local_ny=5; after, rank 0 reports 5 and rank 1reports 0.