Skip to content

Skip a UVM-Heavy Test Under Compute Sanitizer - #1400

Merged
leofang merged 1 commit into
NVIDIA:mainfrom
Andy-Jost:disable-tests-for-release
Dec 18, 2025
Merged

Skip a UVM-Heavy Test Under Compute Sanitizer#1400
leofang merged 1 commit into
NVIDIA:mainfrom
Andy-Jost:disable-tests-for-release

Conversation

@Andy-Jost

@Andy-Jost Andy-Jost commented Dec 17, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Adjust CI to improve debuggability of long-running tests.

Changes

  • Add --durations=0 to CI pytest invocations so logs report per-test timings.
  • Skip cuda_core/tests/test_helpers.py::test_patterngen_seeds when running under compute-sanitizer (detected via cuda_python_test_helpers.under_compute_sanitizer()), since this test can become extremely slow under sanitizer/UVM-heavy behavior.

Rationale

test_patterngen_seeds performs many syncs and host/device-visible memory comparisons; under compute-sanitizer this can balloon to very long runtimes in some CI environments. Skipping it under sanitizer keeps the CI signal while avoiding hour-scale jobs.

Test plan

  • CI

@Andy-Jost Andy-Jost self-assigned this Dec 17, 2025
@copy-pr-bot

copy-pr-bot Bot commented Dec 17, 2025

Copy link
Copy Markdown
Contributor

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@Andy-Jost Andy-Jost changed the title Skip test_latchkernel Skip tests in preparation for release Dec 17, 2025
@Andy-Jost
Andy-Jost marked this pull request as draft December 17, 2025 20:02
@Andy-Jost

Andy-Jost commented Dec 17, 2025

Copy link
Copy Markdown
Contributor Author

/ok to test ec93c1249ec517b2c42648d9c03db0959a2eb2a9

@github-actions

This comment has been minimized.

Sign in to view
@leofang leofang added this to the cuda.core beta 10 milestone Dec 17, 2025
@leofang leofang added bug Something isn't working P0 High priority - Must do! test Improvements or additions to tests cuda.core Everything related to the cuda.core module labels Dec 17, 2025
@Andy-Jost
Andy-Jost force-pushed the disable-tests-for-release branch 2 times, most recently from ce6b1b5 to e3183ab Compare December 17, 2025 21:58
@Andy-Jost

Andy-Jost commented Dec 17, 2025

Copy link
Copy Markdown
Contributor Author

/ok to test e3183abb9dc0403ac63e9d868178c50b1ef8cfd8

@Andy-Jost

Andy-Jost commented Dec 17, 2025

Copy link
Copy Markdown
Contributor Author

/ok to test cb86bd13917a38ad4b0cd3d0801644a9fd712fab

@Andy-Jost

Andy-Jost commented Dec 17, 2025

Copy link
Copy Markdown
Contributor Author

/ok to test 8823fccecc33aa5c1517a8f19afba18ff0eef6e3

@Andy-Jost
CI: add pytest durations and skip patterngen seeds under sanitizer
cbe5b2d 
- Add --durations=0 to CI pytest invocations to report per-test timings.
- Add cuda_python_test_helpers.under_compute_sanitizer() and use it to skip
  the UVM-heavy PatternGen seed test under compute-sanitizer.
@Andy-Jost
Andy-Jost force-pushed the disable-tests-for-release branch from 8823fcc to cbe5b2d Compare December 17, 2025 23:42
@Andy-Jost

Andy-Jost commented Dec 17, 2025

Copy link
Copy Markdown
Contributor Author

/ok to test cbe5b2d

Andy-Jost
Andy-Jost commented Dec 17, 2025
View reviewed changes
Comment thread ci/tools/run-tests
@@ -32,7 +32,7 @@ if [[ "${test_module}" == "pathfinder" ]]; then
echo "Running pathfinder tests with " \
"LD:${CUDA_PATHFINDER_TEST_LOAD_NVIDIA_DYNAMIC_LIB_STRICTNESS} " \
"FH:${CUDA_PATHFINDER_TEST_FIND_NVIDIA_HEADERS_STRICTNESS}"
pytest -ra -s -v tests/ |& tee /tmp/pathfinder_test_log.txt
pytest -ra -s -v --durations=0 tests/ |& tee /tmp/pathfinder_test_log.txt

@Andy-Jost Andy-Jost Dec 17, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems reasonable to report test times so that these issues are easier to debug in the future.

@Andy-Jost
Andy-Jost requested a review from leofang December 17, 2025 23:45
@Andy-Jost Andy-Jost changed the title Skip tests in preparation for release Skip a UVM-Heavy Test Under Compute Sanitizer Dec 17, 2025
@Andy-Jost
Andy-Jost marked this pull request as ready for review December 17, 2025 23:46
@Andy-Jost

Andy-Jost commented Dec 18, 2025

Copy link
Copy Markdown
Contributor Author

CI confirms the fix. This test was taking >20min:

CI / Test linux-64 / py3.12, 12.9.1, local, l4 (push)Successful in 6m

@Andy-Jost

Andy-Jost commented Dec 18, 2025

Copy link
Copy Markdown
Contributor Author

Longest durations now reported in the test logs:

============================== slowest durations ===============================
7.42s call     tests/memory_ipc/test_workerpool.py::TestIpcWorkerPool::test_main[3]
6.67s call     tests/memory_ipc/test_workerpool.py::TestIpcWorkerPoolUsingRegistry::test_main[3]
6.64s call     tests/memory_ipc/test_workerpool.py::TestIpcWorkerPoolUsingIPCDescriptors::test_main[3]
6.63s call     tests/memory_ipc/test_workerpool.py::TestIpcWorkerPoolUsingRegistry::test_main[1]
6.44s call     tests/memory_ipc/test_send_buffers.py::TestIpcReexport::test_main[DeviceMR]
6.38s call     tests/memory_ipc/test_workerpool.py::TestIpcWorkerPool::test_main[1]
5.60s call     tests/memory_ipc/test_memory_ipc.py::TestIPCSharedAllocationHandleAndBufferObjects::test_main[DeviceMR]
5.57s call     tests/memory_ipc/test_memory_ipc.py::TestIPCSharedAllocationHandleAndBufferDescriptors::test_main[DeviceMR]
4.50s call     tests/memory_ipc/test_memory_ipc.py::TestIPCMempoolMultiple::test_main[DeviceMR]
4.43s call     tests/memory_ipc/test_send_buffers.py::TestIpcSendBuffers::test_main[1]
4.40s call     tests/memory_ipc/test_workerpool.py::TestIpcWorkerPoolUsingIPCDescriptors::test_main[1]
4.39s call     tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main[DeviceMR]
3.42s call     tests/memory_ipc/test_event_ipc.py::TestEventIpc::test_main[DeviceMR]
3.36s call     tests/memory_ipc/test_event_ipc.py::TestIpcEventProperties::test_main[False-True-True]
3.36s call     tests/memory_ipc/test_errors.py::TestImportWrongMR::test_main[DeviceMR]
3.36s call     tests/memory_ipc/test_send_buffers.py::TestIpcSendBuffers::test_main[3]
3.35s call     tests/memory_ipc/test_errors.py::TestDanglingBuffer::test_main[DeviceMR]
3.35s call     tests/memory_ipc/test_event_ipc.py::TestIpcEventProperties::test_main[False-False-True]
3.35s call     tests/memory_ipc/test_event_ipc.py::TestIpcEventProperties::test_main[False-True-False]
3.35s call     tests/memory_ipc/test_event_ipc.py::TestIpcEventProperties::test_main[True-False-True]
3.35s call     tests/memory_ipc/test_event_ipc.py::TestIpcEventProperties::test_main[True-True-False]
3.35s call     tests/memory_ipc/test_event_ipc.py::TestIpcEventProperties::test_main[False-False-False]
3.34s call     tests/memory_ipc/test_serialize.py::TestObjectSerializationWithMR::test_main[DeviceMR]
3.34s call     tests/memory_ipc/test_serialize.py::TestObjectPassing::test_main[DeviceMR]
3.33s call     tests/memory_ipc/test_event_ipc.py::TestIpcEventProperties::test_main[True-False-False]
3.32s call     tests/memory_ipc/test_event_ipc.py::TestIpcEventProperties::test_main[True-True-True]
3.32s call     tests/memory_ipc/test_memory_ipc.py::TestIpcMempool::test_main[DeviceMR]
3.31s call     tests/memory_ipc/test_errors.py::TestImportBuffer::test_main[DeviceMR]
2.65s call     tests/memory_ipc/test_errors.py::TestAllocFromImportedMr::test_main[DeviceMR]
2.39s call     tests/example_tests/test_basic_examples.py::TestExamples::test_example[/__w/cuda-python/cuda-python/cuda_core/tests/example_tests/../../examples/vector_add.py]
2.35s call     tests/example_tests/test_basic_examples.py::TestExamples::test_example[/__w/cuda-python/cuda-python/cuda_core/tests/example_tests/../../examples/saxpy.py]
2.25s call     tests/example_tests/test_basic_examples.py::TestExamples::test_example[/__w/cuda-python/cuda-python/cuda_core/tests/example_tests/../../examples/memory_ops.py]
1.65s call     tests/test_helpers.py::test_latchkernel

@leofang
leofang enabled auto-merge (squash) December 18, 2025 00:11
@leofang
leofang merged commit 083315c into NVIDIA:main Dec 18, 2025
80 checks passed
@github-actions

github-actions Bot commented Dec 18, 2025

Copy link
Copy Markdown
Doc Preview CI
Preview removed because the pull request was closed or merged.

@Andy-Jost
Andy-Jost deleted the disable-tests-for-release branch December 18, 2025 15:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@leofang leofang leofang approved these changes

Assignees

@Andy-Jost Andy-Jost

Labels

bug Something isn't working cuda.core Everything related to the cuda.core module P0 High priority - Must do! test Improvements or additions to tests

Projects

None yet

Milestone

cuda.core beta 10

Development

Successfully merging this pull request may close these issues.

2 participants

@Andy-Jost @leofang