Skip to content

Commit 5a0ef2e

Browse files
Andy-Jostclaude
andcommitted
test(cuda.bindings): capture thread-locally in tests run on parallel threads
With pooled contexts the worker threads of a test start together; before, serialized context creation kept them apart. Three tests begin a stream capture in global mode, and in that mode a call such as cuGraphInstantiate in one thread fails while another thread is capturing. The host-function graph tests failed this way on the free-threaded rows. Capture in thread-local mode instead; what the tests check does not depend on it. Drop the synchronize at the end of each pooled test for the same reason: it could break a global capture running in another thread. Pooled contexts now behave like the module-scoped ctx fixture, which does not synchronize between tests either. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent e55a371 commit 5a0ef2e

3 files changed

Lines changed: 19 additions & 12 deletions

File tree

‎cuda_bindings/tests/conftest.py‎

Lines changed: 7 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -49,8 +49,11 @@ class _ThreadContextPool:
4949
The pool gives each running thread a context of its own, as before, and
5050
keeps it for the next test. The plugin empties the pool at the end of each
5151
test module, so a context lives no longer than the module-scoped ``ctx``
52-
fixture does in a run without threads. A context that reports an error at
53-
the end of a test is destroyed rather than reused.
52+
fixture does in a run without threads.
53+
54+
The worker threads of a test now run at the same time. Do not add a
55+
synchronizing call between tests here: while another thread captures a
56+
stream in global mode, such a call fails and breaks that capture.
5457
"""
5558

5659
def __init__(self):
@@ -81,18 +84,13 @@ def context(self):
8184
try:
8285
yield device, ctx
8386
finally:
84-
(sync_err,) = cuda.cuCtxSynchronize()
8587
# Pop rather than unset, so that the context that was current
8688
# before the test (on the main thread, for tests that are not run
8789
# in parallel) is current again.
8890
err, _ = cuda.cuCtxPopCurrent()
8991
assert err == cuda.CUresult.CUDA_SUCCESS
90-
if sync_err == cuda.CUresult.CUDA_SUCCESS:
91-
with self._lock:
92-
self._idle.append(entry)
93-
else:
94-
(err,) = cuda.cuCtxDestroy(ctx)
95-
assert err == cuda.CUresult.CUDA_SUCCESS
92+
with self._lock:
93+
self._idle.append(entry)
9694

9795
def clear(self):
9896
"""Destroy the idle contexts. Call only when no test is running."""

‎cuda_bindings/tests/test_cuda.py‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -706,7 +706,10 @@ def host_fn(userData):
706706
err, stream = cuda.cuStreamCreate(0)
707707
assert err == cuda.CUresult.CUDA_SUCCESS
708708
try:
709-
(err,) = cuda.cuStreamBeginCapture(stream, cuda.CUstreamCaptureMode.CU_STREAM_CAPTURE_MODE_GLOBAL)
709+
# Thread-local mode: pytest-run-parallel runs this test in several threads
710+
# at once, and in global mode a call such as cuGraphInstantiate in one
711+
# thread fails while another thread is capturing.
712+
(err,) = cuda.cuStreamBeginCapture(stream, cuda.CUstreamCaptureMode.CU_STREAM_CAPTURE_MODE_THREAD_LOCAL)
710713
assert err == cuda.CUresult.CUDA_SUCCESS
711714
(err,) = cuda.cuLaunchHostFunc(stream, callback, ctypes.addressof(counter))
712715
assert err == cuda.CUresult.CUDA_SUCCESS

‎cuda_bindings/tests/test_cudart.py‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -616,7 +616,10 @@ def test_cudart_cudaStreamGetCaptureInfo():
616616
assert status == cudart.cudaStreamCaptureStatus.cudaStreamCaptureStatusNone
617617

618618
# start capture
619-
(err,) = cudart.cudaStreamBeginCapture(stream, cudart.cudaStreamCaptureMode.cudaStreamCaptureModeGlobal)
619+
# Thread-local mode: pytest-run-parallel runs this test in several threads
620+
# at once, and in global mode a call such as cudaGraphInstantiate in one
621+
# thread fails while another thread is capturing.
622+
(err,) = cudart.cudaStreamBeginCapture(stream, cudart.cudaStreamCaptureMode.cudaStreamCaptureModeThreadLocal)
620623
assertSuccess(err)
621624

622625
# validate that stream is capturing now
@@ -1469,7 +1472,10 @@ def host_fn(userData):
14691472
err, stream = cudart.cudaStreamCreate()
14701473
assertSuccess(err)
14711474
try:
1472-
(err,) = cudart.cudaStreamBeginCapture(stream, cudart.cudaStreamCaptureMode.cudaStreamCaptureModeGlobal)
1475+
# Thread-local mode: pytest-run-parallel runs this test in several threads
1476+
# at once, and in global mode a call such as cudaGraphInstantiate in one
1477+
# thread fails while another thread is capturing.
1478+
(err,) = cudart.cudaStreamBeginCapture(stream, cudart.cudaStreamCaptureMode.cudaStreamCaptureModeThreadLocal)
14731479
assertSuccess(err)
14741480
(err,) = cudart.cudaLaunchHostFunc(stream, callback, ctypes.addressof(counter))
14751481
assertSuccess(err)

0 commit comments

Comments
 (0)