Set CU_CTX_SCHED_BLOCKING_SYNC option in tests.

[Andy-Jost]

By default, CPU threads spin while waiting for GPU synchronization. This changes the option to block instead while testing.

See cuDevicePrimaryCtxSetFlags for details.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[leofang]

Q: Why is this needed? We don't expect our users to do this.

[Andy-Jost]

This is not essential. It just reduces load on the test machines. I was annoyed by the spinning when working with certain multiprocessing tests and happened to come across this option in the docs. It's a 5-minute change so I figured why not.

[Andy-Jost]

/ok to test c301d9ceb070e002ba1e427b773600e0ddf5fcc6

[leofang]

This is not essential. It just reduces load on the test machines.

Then we should not make this change. We need to test the real-life use scenarios. With the upcoming coverage of various Windows driver modes (#985) our test env will get more sensitive to the underlying driver than ever. Changing the global state is not helping if any issue comes up.

[leofang]

I was annoyed by the spinning when working with certain multiprocessing tests

This is why I kept pushing for IPC events so as to enabling stream-ordered IPC tests. Device synchronization is expansive at the library test scale. We should avoid writing tests that require synchronization. We've had enough in the past to learn in a hard way that stream ordering is critical 😅

[leofang]

I'll take a look at the performance issue. @Andy-Jost do you have in mind speeding up the aggregate performance of all tests under cuda_core/tests/memory_ipc?

[Andy-Jost]

Let me clarify: I'm not saying there's any performance issue. The annoyance I ran into was during development, especially while using code.interact when a latch kernel was spinning. My laptop was burning up haha!

I believe most of the wait in memory_ipc is on multiprocessing.Queue or similar objects, so it is already blocking. As far as I know, we don't have any significant CUDA synchronization waits in those tests.

[leofang]

I would be OK if you could guard this change behind a test-time env var (say, CUDA_CORE_TEST_BLOCKING_SYNC) that makes your development workflow faster. I am still concerned about the blocking nature of our tests, though, but let's make your life easier first.

[Andy-Jost]

Sounds good. Latest upload adds an environment variable guard.

[Andy-Jost]

/ok to test b867b3b

[Andy-Jost]

/ok to test e86d5b7

[github-actions]

Doc Preview CI
Preview removed because the pull request was closed or merged.