[BUG]: Windows system search does use PATH, contrary to docs

Is this a duplicate?

I confirmed there appear to be no duplicate issues for this bug and that I agree to the Code of Conduct

Type of Bug

Runtime Error

Component

cuda.pathfinder

Describe the bug

The load_nvidia_dynamic_lib docs say that on Windows the system search step doesn't look at PATH. #1781 was closed on that basis. But in practice it does find DLLs on PATH.
We hit this in nvmath-python CI. CUDA_PATH pointed at CTK 12.8, but CTK 13.4 was also on PATH:

load_nvidia_dynamic_lib("cublasLt")
-> ...\CUDA\v13.4\bin\x64\cublasLt64_13.dll, found_via="system-search"

System search runs before the CUDA_PATH step, so the 13.4 DLL won over the toolkit we'd selected. cutensor (cu12 wheel) then failed to load with Windows error 126, because it needs cublasLt64_12.dll.

How to Reproduce

Small repro, no CUDA needed. It makes the same call pathfinder's system search makes
(LoadLibraryExW(name, NULL, 0) in load_with_system_search), with a dummy DLL that
only lives in a folder on PATH:

import ctypes, os, shutil, subprocess, sys, tempfile
d = tempfile.mkdtemp()
shutil.copy(os.path.join(os.environ["SystemRoot"], "System32", "version.dll"),
            os.path.join(d, "zz_probe.dll"))
child = ("import ctypes; k = ctypes.WinDLL('kernel32');"
         "k.LoadLibraryExW.restype = ctypes.c_void_p;"
         "print(bool(k.LoadLibraryExW('zz_probe.dll', None, 0)))")
env = dict(os.environ, PATH=d + os.pathsep + os.environ["PATH"])
subprocess.run([sys.executable, "-c", child], env=env)  # prints True

Expected behavior

load_nvidia_dynamic_lib("cublasLt") should either skip PATH on as doc claimed on windows, or update the docs

Operating System

Windows

nvidia-smi output

No response

I would like to work on this issue. I can prepare the smallest targeted change in the Windows loader/search-step path, with focused regression coverage for the documented PATH behavior. Please assign the issue if that direction is appropriate; I will wait for maintainer confirmation before starting implementation.

@rwgk

@rwgk could you take a look?

I'll take a look. I'm really surprised. (I want to see if I can recover the agent session that I used while working on #1781.)

— codex on behalf of Ralf —

The report in #3137 is correct about the Windows documentation error. Native Windows DLL search already supports PATH; the explanation adopted in #1795 incorrectly attributed a process-wide restriction to CPython 3.8+.

The forensic reconstruction also supports the decision to abandon the growing manual PATH scanner. The consequential mistake was accepting an unverified explanation of the earlier loading failures and documenting it as general Windows behavior. The original failures were real, but their precise cause remains unresolved.

I retained the detailed analysis and evidence: issue_3137_forensics_20261104T000313Z_a4c80b39_analysis.txt

The supporting files, including relevant Cursor excerpts, contemporaneous issue/PR snapshots, and the CPython source audit, are bundled here:
issue_3137_forensics_20261104T000313Z_a4c80b39_supporting_files.zip

Full details

The investigation covered #3137, #1781, #1506, the reference-only alternative #1794, and #1795; the recovered January and March Cursor sessions; contemporaneous local GitHub snapshots; and official Microsoft documentation and stock CPython source. The saved chat search covered 119 databases. No new Windows runtime experiment was executed for this investigation.

The critical distinction is between Python's own loading defaults and a native API call made through ctypes. For an ordinary unpackaged process whose native loader policy has not been reconfigured:

Operation	Searches environment `PATH`?
`ctypes.WinDLL("example.dll")`, with `winmode` omitted	No, since Python 3.8
`ctypes.WinDLL("example.dll", winmode=0)`	Yes
`kernel32.LoadLibraryExW("example.dll", None, 0)`	Yes

Pathfinder uses the third operation. Creating the ctypes wrapper for kernel32 does not impose the WinDLL constructor's loading defaults on subsequent calls to its functions. The explicit flags-zero argument reaches Windows unchanged. Microsoft includes PATH in the standard search order for unpackaged applications. Microsoft DLL search order, the CPython ctypes implementation, the underlying native load call.

Complete-tree, case-insensitive source searches at CPython 3.8.0 and 3.13.12 found no SetDefaultDllDirectories occurrence. This includes the Windows startup and launcher sources. Python 3.13.12 is the version recorded in the March Windows regression-test output. CPython instead supplies restricted search flags on its own extension-import and default ctypes loading calls. Python's documentation explicitly distinguishes those calls from process-wide loading behavior. Python 3.8 ctypes source, Python 3.13.12 extension-import source, Python documentation.

An embedding host, imported native component, custom Python distribution, or packaged application can establish different behavior. That is a reason to describe the native process policy accurately; it does not support a universal claim that CPython 3.8+ excludes PATH from raw flags-zero loading.

The accompanying assertion that AddDllDirectory() directories necessarily participate in this raw search is also conditional. They participate when the load requests LOAD_LIBRARY_SEARCH_USER_DIRS, including through LOAD_LIBRARY_SEARCH_DEFAULT_DIRS, or when process defaults have been configured to include them. Microsoft AddDllDirectory documentation.

The recovered history identifies the following sequence.

January: error 126 was overinterpreted as proof of a missing dependency. In Cursor session 600c0747-45cd-49c5-83c0-5161651fcebd, titled PR Code Trace, Ralf showed that nvrtc64_130_0.dll existed in CUDA 13.1's bin\x64 directory and that the directory was on PATH. A basename-plus-flags-zero ctypes probe failed with error 126. Claude 4.5 Opus Thinking asserted that this confirmed a missing dependency rather than a missing primary DLL. The error establishes neither distinction. The next successful experiment changed both the basename to an absolute path and the flags to LOAD_WITH_ALTERED_SEARCH_PATH. It therefore did not isolate the cause. This led to [pathfinder] Use LOAD_WITH_ALTERED_SEARCH_PATH for system DLL search on Windows #1506's SearchPathW plus altered-search implementation. The bundled January excerpt preserves the setup, actual PATH, probes, success report, patch, and commit evidence, with original source line numbers; the unsupported assertion is at source line 1122. The original proof of concept is also archived in the PR.
March: the same uncertainty was converted into the opposite unsupported conclusion. The March 19 experiments showed genuine failures across CTK libraries on main without CUDA_HOME/CUDA_PATH, and success with [pathfinder] Use LOAD_WITH_ALTERED_SEARCH_PATH for system DLL search on Windows #1506. Renaming nvrtc-builtins64_132.dll did not prevent loading nvrtc on the PR branch. Those observations weakened one proposed dependency explanation, but did not establish that native search had failed to discover every primary DLL. In the subsequent analysis, Claude 4.6 Opus treated a null native handle as proof that the primary DLL had never been found, rather than found but unloadable, and introduced the false process-wide CPython explanation.
The independent review inherited that premise. In Cursor session 4a5381c6-1845-41f8-a7e0-dc6f3eb2926a, titled PR 1506 Review, GPT-5.4 Extra High consulted Microsoft API documentation but did not verify the alleged CPython startup behavior against Python documentation or source. Its objection to using SearchPathW had a sound independent basis: Microsoft explicitly discourages using it to locate DLLs for LoadLibrary, because the search semantics differ. However, parts of the review overstated what was established. Ordinary native flags-zero search already includes the current directory, and AddDllDirectory support in that search depends on process configuration. The review did not establish that these were new regressions in the way it claimed. Microsoft's SearchPathW warning.
The manual alternative was correctly recognized as a separate search policy. #1794 preserved native loading first, then manually split PATH, stripped quotes, expanded variables, rejected relative and missing directories, deduplicated directories, excluded driver libraries, and loaded discovered files through the absolute-path helper. In March source record 42, Ralf explicitly objected that the result looked like an emulation rather than an actual system search. The context was that there had been no user complaints, the NVIDIA installer normally sets CUDA_PATH, and removing it for multi-CTK testing was an atypical development workflow. That concern and the agents' responses are also preserved publicly. The alternative's reported 898 passing Linux tests did not validate the Windows semantics: its new Windows-specific tests were skipped.
The decision to stop implementation carried the false premise into documentation. The agents recommended documentation-only changes while presenting the alleged CPython exclusion as established intentional hardening. [doc-only] pathfinder: clarify Windows DLL search semantics #1795 propagated that claim across four files' docstrings. Its reported 897 passing tests were Linux tests and could not establish the Windows claim. [doc-only] pathfinder: clarify Windows DLL search semantics #1795 was merged on March 20, closing [BUG][pathfinder] Windows system search fails to find any CUDA DLL when CUDA_HOME/CUDA_PATH are not set #1781 as working as intended; [pathfinder] Use LOAD_WITH_ALTERED_SEARCH_PATH for system DLL search on Windows #1506 and ONLY FOR REFERENCE, NOT FOR MERGING: pathfinder_iter_env_path_directories branch (on top of PR #1506) #1794 were closed unmerged. Consequently, [doc-only] pathfinder: clarify Windows DLL search semantics #1795 did not remove PATH from native search. It documented an exclusion that the loader code did not enforce. The merged documentation.

The contemporaneous snapshots preserve another useful distinction: the original #1506 body explicitly said the Cursor-generated fix still needed Ralf's own review, and the original #1781 title/body described a dependency-loading failure. Later edits replaced that hypothesis with the global PATH-exclusion diagnosis. The observations did not become more discriminating when the prose became more confident. These superseded versions are included in the ZIP.

The recorded failure messages did not establish where loading failed. LoadLibraryExW can return NULL when the primary DLL or a dependency cannot be loaded. Pathfinder's system-search loop discarded the native error. The eventual No such file: nvrtc*.dll text came from other filesystem search steps, rather than from Windows' native loader. The exact March descriptor already included nvrtc64_130_0.dll, matching the CTK 13.2 filename, so a missing filename-table entry does not explain that result either. Native search loop, final filesystem diagnostic, historical descriptor, Microsoft load-failure semantics.

There is a narrower API limitation relevant to the decision to stop: Windows has no documented LOAD_LIBRARY_SEARCH_PATH flag that simply adds environment PATH to the modern restricted search flags. Combining AddDllDirectory directories with PATH in a chosen order while filtering out the current directory is a deliberate search policy. The manual scanner was implementing such a policy. However, ordinary native search including PATH already existed and was already being called. Safe DLL search mode changes the current directory's position; it does not remove that directory or PATH. SearchPathW has a separate SafeProcessSearchMode, so the two should not be conflated. Microsoft DLL search order, Microsoft SearchPathW documentation.

#3137 exposes a separate toolkit-selection problem. The reported CTK 13.4 versus 12.8 outcome matches the code: system search runs before the CUDA_PATH step, and the candidate names prefer cublasLt64_13.dll before cublasLt64_12.dll. A loaded CUDA 13 DLL cannot satisfy an import for the distinct CUDA 12 basename. Merely setting CUDA_PATH therefore does not guarantee priority over native search in this implementation; rearranging PATH alone can also be insufficient when different major-version basenames are tried in a fixed order. Search sequence, DLL preference order.

The no-CUDA probe in #3137 is sound evidence for its narrow point. It copies version.dll under a unique basename and loads it in a fresh child using the same raw flags-zero call. The reported success matches Microsoft's documented semantics. The renamed basename avoids an obvious already-loaded or KnownDLL false positive. Checking the returned module's filename and adding a PATH-omitted negative control would strengthen the probe, but no apparent flaw defeats its conclusion that native search can find a DLL through PATH. It does not diagnose the old CTK failures or itself reproduce the mixed-version dependency failure.

The precise January/March failure mechanism remains unresolved. January's transcript preserves a PATH dump and native error 126; the later March evidence does not preserve equivalent per-candidate native errors or enough process-loader configuration to identify the cause. Neither investigation recorded a controlled comparison that isolated filename selection from loading flags. Dependency failures, a different process policy, packaging/distribution differences, and environment/search-order details remain possibilities rather than established explanations.

If that historical failure needs to be revisited, the discriminating experiment is a fresh Windows child for each case, keeping the interpreter and environment constant: a uniquely named neutral DLL with and without its directory on PATH; the CTK basename with flags zero; the same absolute filename with flags zero; and the same absolute filename with either LOAD_LIBRARY_SEARCH_DEFAULT_DIRS | LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR or LOAD_WITH_ALTERED_SEARCH_PATH. Immediate GetLastError values and successful GetModuleFileNameW results would preserve the evidence that the original comparison lacked.

The separate March Claude 4.6 chat was not recovered from the backup. Its relevant prompt survives in the March review session and its analysis is preserved in #1506's comments. The bundled excerpts and source audit distinguish recorded observations from agent explanations; they do not claim a new reproduction of the original failure.

A documentation correction is justified. Making CUDA_PATH authoritative, excluding PATH, or guaranteeing consistent versions across dependency loads would each be a separate behavior decision. Reviving either abandoned "restore PATH" patch would not address #3137's complaint: PATH already participates and can win.

My prompt

Now to:

What should we do?

Just correcting the documentation?

I believe I want to revisit the search order, to see if we need to tweak it, so:

That Linux and Windows are aligned as much as possible / within reason.
The rationale for the search order is sound. (hoping that we don't have to revisit Linux)

Could you please thoroughly analyze and give me your recommendation?

codex GPT-6.1-Sol ultra

I recommend keeping the existing default search order on both platforms, correcting the documentation, and adding focused regression coverage. I would address the mixed-version dependency failure separately. I don’t see sufficient justification for changing Linux or introducing a Windows-only CUDA_PATH priority rule.

The strongest new evidence is that the current order was explicitly debated and chosen—not inherited accidentally.

Full details

In PR #604, Leo described this development workflow:

CUDA_HOME points to a complete toolkit.
LD_LIBRARY_PATH exposes locally built math libraries.
Those local builds must take precedence over the toolkit copies.

You raised the opposite concern: someone installs a newer toolkit, sets CUDA_HOME, and is surprised when an older system installation wins. After discussing both cases, you put CUDA_HOME last. That was a deliberate compromise favoring Python package environments and native runtime configuration.

That rationale still holds.

The existing effective priority has a coherent meaning:

Priority	Source	Rationale
0	Already-loaded libraries	Respect existing process state
1	NVIDIA Python wheels	Prefer dependencies installed for the Python application
2	Activated Conda environment	Prefer environment-owned dependencies over system installations
3	Native OS loader	Honor the process’s runtime library configuration
4	Product-specific roots, then `CUDA_PATH` / `CUDA_HOME`	Find additional installations not exposed through earlier mechanisms
5	Configured Program Files locations and canary discovery	Provide additional installation discovery where applicable

That is the shared selection precedence implemented by the current orchestrator. Filesystem discovery happens before the already-loaded check internally, but the already-loaded result takes precedence.

The wheel/Conda priorities also have direct justification: #1003 introduced explicit Conda discovery to prevent an active environment from losing to an unrelated system toolkit, while preserving wheel priority.

One clarification belongs prominently in the documentation:

CUDA_PATH and CUDA_HOME provide fallback installation locations. They do not select or pin the runtime toolkit.

The later change making CUDA_PATH higher priority than CUDA_HOME only established precedence between those two variables. It did not put either above wheels, Conda, or native loading. #1801.

Linux and Windows are already aligned at this policy level. Their native loaders necessarily differ:

Linux delegates to dlopen, which incorporates native rules involving LD_LIBRARY_PATH, RPATH/RUNPATH, the linker cache, and standard directories.
Windows delegates to LoadLibraryExW(..., 0), which normally includes PATH.

Delegating to each native loader preserves platform conventions without implementing another search engine. Exact dependency-resolution behavior will remain platform-specific, including when the primary library is loaded through an absolute filename. Linux loading rules, Windows loading rules.

I would not promote CUDA_PATH on Windows merely because PATH is broader than LD_LIBRARY_PATH. That distinction is real, but it doesn’t reliably identify user intent.

The reported configuration represents one legitimate intention:

CUDA_PATH selects toolkit 12.
An unrelated toolkit 13 remains on PATH.

Another legitimate configuration represents the opposite intention:

CUDA_PATH points to the installed toolkit.
PATH deliberately exposes replacement or development DLLs.

A Windows-only reorder exchanges which configuration surprises its user. Furthermore, CUDA_PATH itself is installer-managed and can be stale. NVIDIA’s Windows installation guide explicitly tells users to revalidate it after toolkit installation or removal. Installation guide.

Promoting it on both platforms would also knowingly reverse the Linux workflow that motivated the existing design.

There is another practical consequence: filesystem discovery finds a candidate before proving it loadable. Once that candidate is selected, an absolute-path load failure raises an error; it doesn’t generally resume native discovery. An early, incompatible toolkit root could therefore prevent a native load that currently succeeds.

Changing Windows to LOAD_LIBRARY_SEARCH_DEFAULT_DIRS would be a larger change still. It would replace the native search domain, affect dependency lookup, remove existing PATH discovery, and introduce different treatment of AddDllDirectory locations. I would not manufacture the behavior claimed by the incorrect documentation.

The more consequential limitation is version selection, which a source-order change cannot fully solve.

Pathfinder makes two separate ordering decisions:

Which source to search.
Which library basename or SONAME to request first.

The second decision can defeat an intuitive interpretation of the first. For example:

PATH contains CUDA 12 before CUDA 13

Try cublasLt64_13.dll → found in CUDA 13
Try cublasLt64_12.dll → never reached

There is an analogous Linux case: LD_LIBRARY_PATH contains CUDA 12, but the linker cache exposes libcublasLt.so.13. If pathfinder requests .so.13 first, the loader can find it in the cache before pathfinder ever requests .so.12.

Native directory precedence applies within the requested name. It does not choose which name pathfinder should request.

This limitation was already discussed under #839. #921 explicitly described newest-first ordering as a heuristic rather than a general solution.

That matters directly to #3137. The current cuTENSOR descriptor requests the dependency "cublasLt" without expressing its required ABI, while the cublasLt descriptor prefers the CUDA 13 name. Dependency declaration.

Moving CUDA_PATH earlier might make that particular fresh-process configuration succeed. It would still leave these cases:

An installed wheel or already-loaded library takes precedence.
A dependency is missing from the specified root and falls back elsewhere.
A consumer requires a different ABI from the registry’s preferred candidate.
Dependencies loaded by the OS follow their own resolution rules.

The desired property is dependency compatibility. Files coming from the same directory or toolkit root is neither a complete guarantee nor always necessary.

There is already relevant work to build on. #1038 covers scoped search, and #654 explicitly discusses satellite-library constraints such as cuTENSOR builds for CUDA 12 versus CUDA 13.

Full search narrowing was subsequently deferred because of its complexity. The active draft #2506 instead adds compatibility checks. It preserves raw search and checks results after loading; it does not select and retry a compatible alternative. Its current definitive metadata coverage also excludes cuTENSOR. I would therefore treat it as related ongoing work, without claiming it already solves #3137. Current draft implementation.

My recommended next steps are:

Correct the documentation now. Cover Windows PATH participation, conditional AddDllDirectory behavior, native-before-root precedence, and the fact that root variables are discovery hints. Annotate the incorrect 1.4.4 release-note claim as well.
Add focused coverage for that contract. One real Windows subprocess test should prove PATH discovery using a uniquely named DLL, a negative control, and verification of the returned filename. Shared orchestration coverage should establish native-before-root precedence while preserving wheel/Conda priority.
Resolve the nvmath CI configuration explicitly. Verify that its installed packages and runtime search environment expose the required CUDA 12 dependencies. Setting CUDA_PATH alone is insufficient. The documentation fix should not be presented as resolving that runtime failure.
Use the existing compatibility work for the stronger requirement. Specify the cu12 consumer/cu13 installation scenario concretely before changing APIs or defaults. I would not introduce another priority switch or revive full search scoping as part of this correction.

This leaves Linux behavior unchanged, preserves a defensible shared policy, and gives the mixed-version failure a more precise follow-up than changing which ambiguous environment setting wins.

This assessment is based on source, design history, and platform documentation; I made no code changes or new runtime experiments.

Navigation Menu

Repository navigation

[BUG]: Windows system search does use PATH, contrary to docs #3137

Is this a duplicate?

Type of Bug

Component

Describe the bug

How to Reproduce

Expected behavior

Operating System

nvidia-smi output

My prompt

codex GPT-6.1-Sol ultra

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[BUG]: Windows system search does use PATH, contrary to docs #3137

Description

Is this a duplicate?

Type of Bug

Component

Describe the bug

How to Reproduce

Expected behavior

Operating System

nvidia-smi output

Activity

nanookclaw commented on Oct 17, 2026

leofang commented on Nov 3, 2026

rwgk commented on Nov 3, 2026

rwgk commented on Nov 4, 2026

rwgk commented on Nov 4, 2026

My prompt

codex GPT-6.1-Sol ultra

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions