Skip to content

Avoid validating every sibling modulefile when resolving an exact version #658

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
xdelaruelle merged 1 commit into from
Jul 29, 2026
Merged

Avoid validating every sibling modulefile when resolving an exact version #658

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .hunspell.en.dic
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1312,6 +1312,9 @@ profiler
moduleTop
struct
load2
load3
ms
stopmod
list2
lrem
replaceFromList
Expand Down
33 changes: 28 additions & 5 deletions MIGRATING.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,21 +38,44 @@ have been implemented:
match the module root name specification, instead of comparing against all
loaded modules.

* Lighter-weight resolution of an exact ``name/version`` specification: sites
with a modulepath directory holding hundreds or thousands of sibling
modulefiles reported slow ``module load`` commands even when the requested
version already exists. Correctly computing the implicit ``default`` and
``latest`` symbolic versions of such a directory used to require validating
every sibling modulefile in it. The lookup now walks candidate versions from
the dictionary-highest to the dictionary-lowest and stops validating as soon
as this can be answered: either a higher valid version is found (the
requested one is then known not to be ``default`` or ``latest``), or the
requested version itself is reached, in which case
nothing higher exists. This bypasses the exhaustive scan without changing
the resolved result.

For this work, the :command:`mb` utility was extended with new benchmark tests
that loads 137 modulefiles, lists the loaded modules, and then purges them.
Running this benchmark highlights the performance improvements, as shown in
the table below:
that loads 137 modulefiles (``load2``), lists the loaded modules (``list2``),
and then purges them. Running this benchmark highlights the performance
improvements, as shown in the table below:

+---------+-------------+--------------------+
| | v5.6.1 | v5.7.0 |
+=========+=============+====================+
| load | 1038 ms | 384 ms (-63%) |
| load2 | 1038 ms | 384 ms (-63%) |
+---------+-------------+--------------------+
| list | 222 ms | 29 ms (-87%) |
| list2 | 222 ms | 29 ms (-87%) |
+---------+-------------+--------------------+
| purge | 568 ms | 303 ms (-46%) |
+---------+-------------+--------------------+

A ``load3`` benchmark test was also added, loading a low, non-default
version of a module out of 500 sibling versions in the same directory. It
specifically exercises the lighter-weight resolution described above:

+---------+-------------+--------------------+
| | v5.6.1 | v5.7.0 |
+=========+=============+====================+
| load3 | 30 ms | 19 ms (-36%) |
+---------+-------------+--------------------+

These measurements were obtained on a system where all modulefiles are stored
on local flash storage. If your modulefiles reside on a shared network
filesystem, using a `Module cache`_ is strongly recommended.
Expand Down
7 changes: 7 additions & 0 deletions NEWS.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,13 @@ Modules 5.7.0 (not yet released)
prepends and the entries it appends, so this entry is not silently
absorbed by these commands' default de-duplication behavior nor relocated
by the :mconfig:`path_entry_reorder` configuration option.
* Resolve an exact ``name/version`` specification without validating every
sibling modulefile found in its directory, by walking versions from the
dictionary-highest to the dictionary-lowest and stopping as soon as the
implicit ``default`` and ``latest`` symbolic versions are settled. A
substantial performance improvement is observed when loading an existing
modulefile out of a directory holding hundreds or thousands of sibling
versions. (fix issue #561)


.. _5.6 release notes:
Expand Down
79 changes: 72 additions & 7 deletions script/mb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
#!/usr/bin/env tclsh
#
# MB, make bench between modulecmd versions
# Copyright (C) 2019-2025 Xavier Delaruelle
# Copyright (C) 2019-2026 Xavier Delaruelle
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
Expand All @@ -27,7 +27,7 @@ Available tests:
help, avail, avail_cache, avail2, avail2_cache, avail3, avail3_cache,
avail4, avail4_cache, whatis, whatis_cache, whatis2, whatis2_cache,
whatis3, whatis3_cache, apropos, load, load_cache, load2, load2_cache,
list, list2, purge, unload
load3, load3_cache, list, list2, purge, unload
(all tests selected by default)

Modes:
Expand Down Expand Up @@ -196,7 +196,8 @@ foreach arg $argv {
help - avail - avail_cache - avail2 - avail2_cache - avail3 -\
avail3_cache - avail4 - avail4_cache - whatis - whatis_cache -\
whatis2 - whatis2_cache - whatis3 - whatis3_cache - apropos - load -\
load_cache - load2 - load2_cache - list - list2 - purge - unload {
load_cache - load2 - load2_cache - load3 - load3_cache -\
list - list2 - purge - unload {
lappend testlist $arg
}
--with-lmod=* {
Expand Down Expand Up @@ -232,7 +233,7 @@ if {![info exists testlist]} {
set testlist [list help avail avail_cache avail2 avail2_cache avail3\
avail3_cache avail4 avail4_cache whatis whatis_cache whatis2\
whatis2_cache whatis3 whatis3_cache apropos load load_cache load2\
load2_cache list list2 purge unload]
load2_cache load3 load3_cache list list2 purge unload]
}
} elseif {[info exists lmod_cmd]} {
foreach test $testlist {
Expand Down Expand Up @@ -338,12 +339,70 @@ catch {unset env(__MODULES_SHARE_MODULEPATH)}
# define bench tests and their arguments and environment
set modpath $curdir/testsuite/modulefiles
array set testsubcmdlist [list avail2 avail avail3 avail avail4 avail whatis2\
whatis whatis3 whatis load2 load list2 list]
whatis whatis3 whatis load2 load load3 load list2 list]
array set testarglist [list avail2 load whatis2 load load load/all unload\
load/all load2 R-bundle-Bioconductor/3.18-foss-2023a-R-4.3.2]
load/all load2 R-bundle-Bioconductor/3.18-foss-2023a-R-4.3.2 load3\
bigmod/1.1]
array set testoptlist [list avail4_lmod --ignore_cache load2_lmod\
--ignore_cache]

# the load3 bench needs a module root holding several hundred sibling
# versions (mixing valid and invalid modulefiles, matching the shape
# reported by sites with large modulepath directories), generated on the
# fly rather than checked into the repository and removed once done
set bigmoddir $modpath.bigmod/bigmod
set need_bigmod [expr {{load3} in $testlist || {load3_cache} in $testlist}]
if {$need_bigmod} {
file mkdir $bigmoddir
set bigmod_content {#%Module

proc ModulesHelp {} {
puts stderr {Lorem ipsum dolor sit amet, consectetur adipiscing elit.}
puts stderr {Sed non risus. Suspendisse lectus tortor, dignissim sit amet,}
puts stderr {adipiscing nec, ultricies sed, dolor. Cras elementum ultrices diam.}
puts stderr {Maecenas ligula massa, varius a, semper congue, euismod non, mi.}
puts stderr {Proin porttitor, orci nec nonummy molestie, enim est eleifend mi,}
puts stderr {non fermentum diam nisl sit amet erat. Duis semper. Duis arcu massa,}
puts stderr {scelerisque vitae, consequat in, }
puts stderr {Lorem ipsum dolor sit amet, consectetur adipiscing elit.}
puts stderr {Sed non risus. Suspendisse lectus tortor, dignissim sit amet,}
puts stderr {adipiscing nec, ultricies sed, dolor. Cras elementum ultrices diam.}
puts stderr {Maecenas ligula massa, varius a, semper congue, euismod non, mi.}
puts stderr {Proin porttitor, orci nec nonummy molestie, enim est eleifend mi,}
puts stderr {non fermentum diam nisl sit amet erat. Duis semper. Duis arcu massa,}
puts stderr {scelerisque vitae, consequat in, }
puts stderr {Lorem ipsum dolor sit amet, consectetur adipiscing elit.}
puts stderr {Sed non risus. Suspendisse lectus tortor, dignissim sit amet,}
puts stderr {adipiscing nec, ultricies sed, dolor. Cras elementum ultrices diam.}
puts stderr {Maecenas ligula massa, varius a, semper congue, euismod non, mi.}
puts stderr {Proin porttitor, orci nec nonummy molestie, enim est eleifend mi,}
puts stderr {non fermentum diam nisl sit amet erat. Duis semper. Duis arcu massa,}
puts stderr {scelerisque vitae, consequat in, }
}

# comment code of the modulefile
module-whatis {returns all various module-info values}

# comment code of the modulefile
setenv FOO value
append-path BAR /path/to/directory
setenv QUX {Lorem ipsum dolor sit amet, consectetur adipiscing elit.}
setenv FOO2 value
append-path BAR2 /path/to/directory
setenv QUX2 {Lorem ipsum dolor sit amet, consectetur adipiscing elit.}
}
for {set bigmodvers 1} {$bigmodvers <= 500} {incr bigmodvers} {
set fid [open $bigmoddir/1.$bigmodvers w]
puts $fid $bigmod_content
close $fid
}
foreach bigmodvers {50 150 250 350 450} {
set fid [open $bigmoddir/1.$bigmodvers.orig w]
puts $fid {not a modulefile}
close $fid
}
}

# load test environment recorded in external files
foreach testenv_file [glob $curdir/testsuite/mb/testenv_*] {
set test [string range [file tail $testenv_file] 8 end]
Expand All @@ -368,7 +427,8 @@ set testenvlist(purge_lmod) $testenvlist(list2_lmod)
# some tests have a minimum version requirement
array set testcompatlist [list avail3 4.6.0 whatis3 4.6.0 avail_cache 5.3.0\
avail2_cache 5.3.0 avail3_cache 5.3.0 whatis_cache 5.3.0 whatis2_cache\
5.3.0 whatis3_cache 5.3.0 load_cache 5.3.0 load2_cache 5.3.0]
5.3.0 whatis3_cache 5.3.0 load_cache 5.3.0 load2_cache 5.3.0 load3_cache\
5.3.0]

# adapt output table to test mode
if {$mode eq {profile}} {
Expand Down Expand Up @@ -424,6 +484,11 @@ foreach tag $taglist {
catch {file delete modulecmd.$tag lib/libtclenvmodules.so.$tag}
}
}

# clean generated load3 bench modulefiles
if {$need_bigmod} {
file delete -force $modpath.bigmod
}
catch {file delete tcl/mfinterp.tcl}

# restore workspace if saved
Expand Down
136 changes: 130 additions & 6 deletions tcl/modfind.tcl.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2984,10 +2984,112 @@ proc findModulesInMemCache {searchid} {
return [list $match_searchid $mod_list]
}

# when 'mod_name' is 'stop_mod''s own parent directory, narrow
# 'elt_list_name' (in the {path hid} pairs shape returned by
# getFilesInDirectory) down to only what is needed to tell whether stop_mod
# is the implicit 'default'/'latest' version of that directory: its
# dotfiles (always kept, the general consolidation phase in getModules
# deals with them normally) plus, among its plain elements, only the
# dictionary range from stop_mod up to either the first higher element that
# is itself valid and not hidden (stop_mod is then known not to be
# default/latest) or the highest element if none qualifies. Left untouched
# in any other case: this is not stop_mod's own directory, stop_mod does not
# exist in it, or it is not itself a valid modulefile, in which case there
# is nothing to optimize
proc trimElemListForStopmod {dir mod_name stop_mod elt_list_name} {
if {![string length $stop_mod] || $mod_name ne [file dirname $stop_mod]} {
return
}

upvar $elt_list_name elt_list
array set elt_arr $elt_list

if {![info exists elt_arr($dir/$stop_mod)] || [lindex [checkValidModule\
$dir/$stop_mod] 0] ne {true}} {
return
}

# first fill updated element list with rc files if any and remove them from
# reference array to work only on other items
array set up_elt_arr {}
foreach rc_tail {.modulerc .version} {
set elt $dir/$mod_name/$rc_tail
if {[info exists elt_arr($elt)]} {
set up_elt_arr($elt) $elt_arr($elt)
unset elt_arr($elt)
}
}

set tail_list {}
foreach elt [array names elt_arr] {
lappend tail_list [file tail $elt]
}

# source the modulepath root '.modulerc' then this directory's own
# '.modulerc' and '.version' right away (in that order) if not already
# done, as any of them may set a hiding rule needed below, to correctly
# tell if a sibling actually is a qualifying higher element (guarded the
# same way the general consolidation phase guards its own later,
# deferred sourcing of them)
global g_rcfilesSourced
set top_rc_file $dir/.modulerc
if {![info exists g_rcfilesSourced($top_rc_file)] && [file readable\
$top_rc_file]} {
execute-modulerc $top_rc_file .modulerc .modulerc
set g_rcfilesSourced($top_rc_file) 1
}
foreach rc_tail {.modulerc .version} {
set rc_elt $dir/$mod_name/$rc_tail
if {[info exists up_elt_arr($rc_elt)] && ![info exists\
g_rcfilesSourced($rc_elt)]} {
set rc_mod $mod_name/$rc_tail
execute-modulerc $rc_elt $rc_mod $rc_mod
set g_rcfilesSourced($rc_elt) 1
}
}

# sort this directory's plain elements and look, starting from the
# highest one down to stop_mod, for the first element that is itself
# valid and not hidden: whether or not it turns out to be stop_mod
# itself, it is this directory's true implicit default/latest, needed
# as-is by the caller to correctly resolve it for every module in this
# directory, not just stop_mod. Only the elements from that point up to
# the highest one need to be kept to reflect it, plus stop_mod's own
# element if it lies below that point
set stop_tail [file tail $stop_mod]
set sorted_tails [lsort -dictionary $tail_list]
set stop_idx [lsearch -exact $sorted_tails $stop_tail]
set top_idx $stop_idx
for {set ti [expr {[llength $sorted_tails] - 1}]} {$ti > $stop_idx}\
{incr ti -1} {
set fp_tail [lindex $sorted_tails $ti]
set fp_elt $dir/$mod_name/$fp_tail
if {[file isdirectory $fp_elt]} {
# a nested version directory: its own validity cannot be settled
# without recursing into it, give up trying to stop early, keep
# everything untouched like usual
return
}
if {[lindex [checkValidModule $fp_elt] 0] eq {true} &&\
![isModuleHidden $mod_name/$fp_tail $stop_mod 0 $fp_elt]} {
set top_idx $ti
break
}
}

# only keep rc files, stop_mod and highest element the updated structure
set up_elt_arr($dir/$stop_mod) $elt_arr($dir/$stop_mod)
if {$top_idx > $stop_idx} {
set elt $dir/$mod_name/[lindex $sorted_tails $top_idx]
set up_elt_arr($elt) $elt_arr($elt)
}
set elt_list [array get up_elt_arr]
}

# Walk through provided list of directories and files to find modules
proc findModulesFromDirsAndFiles {dir full_list depthlvl fetch_mtime\
res_arrname {indir_arrname {}} {hidden_listname {}} {fknown_arrname {}}\
{dknown_arrname {}}} {
{dknown_arrname {}} {stop_mod {}}} {
# link to variables/arrays from upper context
upvar $res_arrname mod_list
if {$indir_arrname ne {}} {
Expand Down Expand Up @@ -3024,6 +3126,12 @@ proc findModulesFromDirsAndFiles {dir full_list depthlvl fetch_mtime\
set mod_list($modulename) [list accesserr [parseAccessIssue\
$element] $element]
} else {
# narrow elt_list down to what is needed to correctly tell
# whether stop_mod is the implicit default/latest version of
# this directory, when this is stop_mod's own directory
# (left untouched in any other case)
trimElemListForStopmod $dir $modulename $stop_mod elt_list

# Add each element in the current directory to the list
foreach {fpelt hid} $elt_list {
lappend full_list $fpelt
Expand Down Expand Up @@ -3082,13 +3190,15 @@ proc findModulesFromDirsAndFiles {dir full_list depthlvl fetch_mtime\
}
}

# finds all module-related files matching mod in the module path dir
proc findModules {dir mod depthlvl fetch_mtime} {
# finds all module-related files matching mod in the module path dir.
# 'stop_mod' helps optimize the search when looking for an exact existing
# modulefile
proc findModules {dir mod depthlvl fetch_mtime {stop_mod {}}} {
reportDebug "finding '$mod' in $dir (depthlvl=$depthlvl,\
fetch_mtime=$fetch_mtime)"

# generated search id (for cache search/save) by compacting given args
set searchid $dir:$mod:$depthlvl:$fetch_mtime
set searchid $dir:$mod:$depthlvl:$fetch_mtime:$stop_mod

# look at memory cache for a compatible result
lassign [findModulesInMemCache $searchid] cache_searchid cache_list
Expand Down Expand Up @@ -3127,7 +3237,8 @@ proc findModules {dir mod depthlvl fetch_mtime} {
}

# walk through list of dirs and files to find modules
findModulesFromDirsAndFiles $dir $full_list $depthlvl $fetch_mtime mod_list
findModulesFromDirsAndFiles $dir $full_list $depthlvl $fetch_mtime\
mod_list {} {} {} {} $stop_mod

reportDebug "found [array names mod_list]"

Expand Down Expand Up @@ -3212,6 +3323,18 @@ proc getModules {dir {mod {}} {fetch_mtime 0} {search {}} {filter {}}} {
set add_rc_defs 1
array set found_list {}
} else {
# an exact 'name/version' query (no wildcard/contains, no result
# filtering), possibly along with a variant or an extra match search
# specifier, resolved in a single-module context (unlike a listing
# one, e.g. avail/whatis/search) lets findModules shortcut its
# directory search (see there); left empty in any other case
set stop_mod {}
set modnamevers [getModuleNameAndVersFromVersSpec $mod]
if {$resctx && $earlyfilter && $filter eq {} && [getCmpSpecFromVersSpec\
$mod] eq {eq} && $querydepth == 1} {
set stop_mod $modnamevers
}

# find modules by searching mod root name in order to catch all module
# related entries to correctly computed auto symbols afterward

Expand All @@ -3233,7 +3356,8 @@ proc getModules {dir {mod {}} {fetch_mtime 0} {search {}} {filter {}}} {
# unless EMS need to be performed (findModules should fetch everything)
set depthlvl [expr {$indepth || $ems_required ? 0 : $querydepth + 1}]

array set found_list [findModules $dir $findmod $depthlvl $fetch_mtime]
array set found_list [findModules $dir $findmod $depthlvl\
$fetch_mtime $stop_mod]
}

# Phase #1: consolidate every kind of entries (directory, modulefile,
Expand Down
1 change: 1 addition & 0 deletions testsuite/mb/testenv_load3
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
MODULEPATH $modpath.bigmod
Loading
Loading