Skip to content

Fix startup self-deadlock with a thread-local recursion guard (minimal alt to #2443)#2488

Merged
jeremyfowers merged 2 commits into
mainfrom
startup_deadlock_minimal
Jun 29, 2026
Merged

Fix startup self-deadlock with a thread-local recursion guard (minimal alt to #2443)#2488
jeremyfowers merged 2 commits into
mainfrom
startup_deadlock_minimal

Conversation

@jeremyfowers

@jeremyfowers jeremyfowers commented Jun 29, 2026

Copy link
Copy Markdown
Member

Minimal alternative to #2443 for the startup deadlock (#2414).

Closes #2414

Root cause

get_system_info_with_cache() holds s_system_info_mutex across the whole recipe build. build_recipes_info() can re-enter the function on the same threadbuild_recipes_info → backend install-params (e.g. SdServer) → SystemInfo::get_rocm_arch() → get_system_info_with_cache() — and re-locking the non-recursive mutex self-deadlocks, hanging at "Building models cache...". Triggered on AMD ROCm systems with an sd-cpp/llamacpp backend pinned to _bin = latest.

Fix (25 lines, one file)

A thread_local bool s_building_recipes guard. The re-entrant call returns the already-computed hardware snapshot (all get_rocm_arch() needs) instead of re-locking. thread_local is deliberate: only the recursing thread short-circuits, so genuinely concurrent threads still block on the mutex and receive fully-populated data.

Why this instead of #2443

The only real deadlock here is the same-thread recursion — no path in the recipe-build chain ever acquires models_cache_mutex_, so there is no ABBA between the two mutexes. That lets the fix stay tiny and avoids the trade-offs in #2443:

  • A global s_phase2_in_progress atomic can't tell same-thread recursion from a concurrent caller, so a thread arriving during the cold recipe build skips recipe computation and returns a recipe-less snapshot (FLM reported unsupported; models filtered out as "Recipe not found"). A thread_local guard avoids that race by construction.
  • No need for detect_rocm_arch_from_sysfs() + the hand-maintained PCI-ID→gfx table (a maintenance liability that doesn't even cover the gfx115x targets), the generation counter, or the duplicated _from_cache / (…, si) overloads.

Net: +25/-0 vs #2443's +383/-82.

Testing

  • Builds clean (cmake --build --preset default --target lemond).
  • lemond starts, passes "Building models cache...", /health returns ok, and /api/v1/models returns a populated list (recipes built, nothing wrongly filtered).

🤖 Generated with Claude Code

…uard

get_system_info_with_cache() holds s_system_info_mutex across the whole
recipe build. build_recipes_info() can re-enter the function on the same
thread (build_recipes_info -> backend install-params -> get_rocm_arch ->
get_system_info_with_cache), and re-locking the non-recursive mutex
self-deadlocks, hanging startup at "Building models cache...". Fixes #2414.

Break the recursion with a thread_local flag: the re-entrant call returns
the already-computed hardware snapshot (all get_rocm_arch needs) instead of
re-locking. thread_local is intentional — only the recursing thread
short-circuits, so concurrent threads still block on the mutex and get
fully-populated data rather than a recipe-less snapshot.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@ckuethe

ckuethe commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Works on both my machines that were previously experiencing the deadlock

@jeremyfowers jeremyfowers added this to the Lemonade v10.9 milestone Jun 29, 2026
@jeremyfowers jeremyfowers marked this pull request as ready for review June 29, 2026 18:55
@github-actions github-actions Bot added bug Something isn't working cpp runtime::rocm AMD ROCm runtime labels Jun 29, 2026
@jeremyfowers jeremyfowers requested a review from fl0rianr June 29, 2026 19:03

@fl0rianr fl0rianr left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. This is a much cleaner targeted fix than broad lock reordering or duplicating cache-aware overloads.

One small non-blocking suggestion: consider adding an assert(s_hardware_computed) or a short comment near the early return to document that this path is only safe because the guard is set after hardware detection?

Per review: the s_building_recipes flag is only set after hardware detection,
so the early-return snapshot always has "devices" populated. Make that
ordering dependency explicit in the comment.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jeremyfowers jeremyfowers enabled auto-merge June 29, 2026 19:30
@jeremyfowers jeremyfowers added this pull request to the merge queue Jun 29, 2026
Merged via the queue into main with commit 283ed43 Jun 29, 2026
77 checks passed
@jeremyfowers jeremyfowers deleted the startup_deadlock_minimal branch June 29, 2026 21:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working cpp runtime::rocm AMD ROCm runtime

Projects

None yet

Development

Successfully merging this pull request may close these issues.

deadlock in get_system_info_with_cache prevents proper lemond startup

3 participants