Fix startup self-deadlock with a thread-local recursion guard (minimal alt to #2443)#2488
Merged
Conversation
…uard get_system_info_with_cache() holds s_system_info_mutex across the whole recipe build. build_recipes_info() can re-enter the function on the same thread (build_recipes_info -> backend install-params -> get_rocm_arch -> get_system_info_with_cache), and re-locking the non-recursive mutex self-deadlocks, hanging startup at "Building models cache...". Fixes #2414. Break the recursion with a thread_local flag: the re-entrant call returns the already-computed hardware snapshot (all get_rocm_arch needs) instead of re-locking. thread_local is intentional — only the recursing thread short-circuits, so concurrent threads still block on the mutex and get fully-populated data rather than a recipe-less snapshot. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
|
Works on both my machines that were previously experiencing the deadlock |
fl0rianr
approved these changes
Jun 29, 2026
fl0rianr
left a comment
Collaborator
There was a problem hiding this comment.
Looks good to me. This is a much cleaner targeted fix than broad lock reordering or duplicating cache-aware overloads.
One small non-blocking suggestion: consider adding an assert(s_hardware_computed) or a short comment near the early return to document that this path is only safe because the guard is set after hardware detection?
Per review: the s_building_recipes flag is only set after hardware detection, so the early-return snapshot always has "devices" populated. Make that ordering dependency explicit in the comment. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Minimal alternative to #2443 for the startup deadlock (#2414).
Closes #2414
Root cause
get_system_info_with_cache()holdss_system_info_mutexacross the whole recipe build.build_recipes_info()can re-enter the function on the same thread —build_recipes_info → backend install-params (e.g. SdServer) → SystemInfo::get_rocm_arch() → get_system_info_with_cache()— and re-locking the non-recursive mutex self-deadlocks, hanging at "Building models cache...". Triggered on AMD ROCm systems with an sd-cpp/llamacpp backend pinned to_bin = latest.Fix (25 lines, one file)
A
thread_local bool s_building_recipesguard. The re-entrant call returns the already-computed hardware snapshot (allget_rocm_arch()needs) instead of re-locking.thread_localis deliberate: only the recursing thread short-circuits, so genuinely concurrent threads still block on the mutex and receive fully-populated data.Why this instead of #2443
The only real deadlock here is the same-thread recursion — no path in the recipe-build chain ever acquires
models_cache_mutex_, so there is no ABBA between the two mutexes. That lets the fix stay tiny and avoids the trade-offs in #2443:s_phase2_in_progressatomic can't tell same-thread recursion from a concurrent caller, so a thread arriving during the cold recipe build skips recipe computation and returns a recipe-less snapshot (FLM reported unsupported; models filtered out as "Recipe not found"). Athread_localguard avoids that race by construction.detect_rocm_arch_from_sysfs()+ the hand-maintained PCI-ID→gfx table (a maintenance liability that doesn't even cover the gfx115x targets), the generation counter, or the duplicated_from_cache/(…, si)overloads.Net: +25/-0 vs #2443's +383/-82.
Testing
cmake --build --preset default --target lemond).lemondstarts, passes "Building models cache...",/healthreturns ok, and/api/v1/modelsreturns a populated list (recipes built, nothing wrongly filtered).🤖 Generated with Claude Code