cuda.core: define the error handling policy and report failures that cannot be raised

Write down how cuda.core handles CUDA failures (docs/source/error_handling.rst
for users, a "Failure handling" section in AGENTS.md and _cpp/DESIGN.md for
contributors) and bring the code into line with it:

- Add cuda.core.CUDAWarning, emitted for CUDA errors that cannot be raised
  (destructors, CUDA callbacks, cleanup after an earlier failure). The C++
  handle layer reports through one helper that uses the Python warnings
  machinery when the interpreter is usable, delivers an escalated warning as an
  unraisable exception, and falls back to stderr otherwise.
  CUDA_ERROR_DEINITIALIZED is not reported.
- Wrap every destroy call made from a deleter (pw_*) so its failure is
  reported instead of discarded, including memory pools, green contexts,
  graphs, graph execs, graphics resources, the linker, user objects, the
  NVRTC/NVVM/nvJitLink handles and file descriptors; release the GIL around
  the compiler-handle destroys like the CUDA ones.
- When the caller's context cannot be restored after a successful operation,
  undo the creation and raise a CUDAError that says which context is current;
  report the same failure as a warning in deleters; report a skipped
  context-sensitive undo instead of leaking silently.
- Add context_get_device and graph_node_set_params so Stream_get_ctx_device
  and _set_definition_node_params stop hand-rolling cuCtxPush/Pop/SetCurrent.
  The node update now publishes its attachment before raising a restoration
  failure, closing a window that left the node referencing released owners.
- Device.set_current(ctx) switches with a single cuCtxSetCurrent, so a failure
  leaves the previous context current and the call works without one.
- Report failed cuStreamEndCapture in GraphBuilder.__dealloc__ and failed
  child-graph rollbacks; warn from _mr_dealloc_callback instead of printing.
- Add a test hook that makes the next context restoration fail, tests for the
  policy, and release notes for 1.3.0.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Author: Andy-Jost

cuda_core/AGENTS.md

@@ -101,6 +101,62 @@ and agents should flag violations.
   (kernel arguments, memcpy/memset operands, `dst_owner`/`src_owner`, and
   host-callback closures) inherit this contract.
 
+## Failure handling
+
+The user-facing contract lives in `docs/source/error_handling.rst`; the rules
+below are for contributors. Reviewers and agents should flag violations.
+
+- **Raise by default**: any failure on a path where an exception can propagate
+  raises. Driver statuses go through `HANDLE_RETURN` (Cython) or are returned as
+  `CUresult` from the C++ handle layer and then `HANDLE_RETURN`ed; never
+  replace a `CUresult` with a generic `RuntimeError`, and drain
+  `get_last_error()` immediately after a handle constructor returns empty so a
+  stale status cannot be misattributed later.
+- **Guarantees**: a call that creates a resource must create nothing when it
+  raises (undo the creation if a later step fails). Every call except
+  `Device.set_current` must leave the calling thread's current context as it
+  found it. Do not hand-roll `cuCtxPush/Pop/SetCurrent` sequences in Cython; use
+  the handle layer's scoped-context helpers (`invoke_in_context`,
+  `invoke_in_context_or_undo`, `cleanup_in_context`, `context_get_device`,
+  `graph_node_set_params`) so the failure handling exists in one place.
+- **Publish before you raise**: when a driver mutation has succeeded and a later
+  step can still fail, commit whatever keeps that mutation memory-safe (for
+  example the graph attachment that retains a node's new owners) before raising
+  the later error. Rolling back the retention of a live mutation creates a
+  dangling reference. When ownership cannot be established, retain the
+  resources anyway (leak) rather than release them; a leak is always preferred
+  to a use-after-free.
+- **Non-propagating paths never raise and never discard a status**: shared_ptr
+  deleters, `__dealloc__`, CUDA callbacks and cleanup after a failure report
+  through one channel, `report_cuda_error()` / `report_message()` in C++ (the
+  `pw_*` wrappers) or `warnings.warn(..., CUDAWarning)` in Cython and Python,
+  which emits `cuda.core.CUDAWarning`. No `print(file=sys.stderr)` and no
+  `fprintf` outside that helper. `CUDA_ERROR_DEINITIALIZED` is filtered by the
+  helper because it means the driver is shutting down.
+- **Rollback failure**: the original exception propagates; the failed rollback
+  is reported out of band (or chained with `raise ... from` when a second
+  exception must be raised). Bare `except:` is acceptable only for
+  rollback-then-`raise` blocks.
+- **Finalization**: once `py_is_finalizing()` is true, do no Python work from
+  destructors or callbacks and accept the leak (see
+  `_cpp/resource_handles.hpp` and `_cpp/GRAPH_ATTACHMENTS.md`).
+- **Aborting**: `std::abort` (or any process termination) is reserved for an
+  internal invariant violation where continuing could corrupt memory or produce
+  silently wrong results *and* no leak-based fallback exists. A failed CUDA
+  call, including a failed context restoration, never qualifies: raise or
+  report instead. There is currently no such path; if one is ever needed it
+  must go through a single helper that writes a diagnostic (call, CUDA error,
+  invariant, "please report") to stderr before aborting, must never trigger
+  during interpreter finalization or for driver-shutdown errors, and must be
+  called out in the docs and release notes. An *implicit* abort (an exception
+  escaping a `noexcept` function or a deleter, including `std::bad_alloc` from
+  an allocation inside `noexcept` code) is a bug (#1489, #2417), not a policy
+  choice: `noexcept` helpers must not allocate, or must catch what they call.
+- **Testing**: inject restoration failures with
+  `cuda.core._resource_handles._set_context_restore_fault_for_testing`; assert
+  reports with `pytest.warns(CUDAWarning)` or `warnings.catch_warnings`, never
+  by matching stderr text.
+
 ## API design guidelines
 
 These are some API design guidelines we try to follow when adding new APIs to

cuda_core/cuda/core/__init__.py

@@ -102,8 +102,10 @@ class _PatchedProperty(metaclass=_PatchedPropMeta):
 from cuda.core._stream import __all__ as _stream_all
 from cuda.core._tensor_map import *
 from cuda.core._tensor_map import __all__ as _tensor_map_all
+from cuda.core._utils.cuda_utils import CUDAWarning
 
 __all__ = [
+    "CUDAWarning",
     *_context_all,
     *_device_all,
     *_device_resources_all,

cuda_core/cuda/core/_cpp/DESIGN.md

@@ -275,6 +275,37 @@ Related functions:
 - `peek_last_error()`: Returns the error without clearing it
 - `clear_last_error()`: Clears the error state
 
+Some functions return a `CUresult` directly instead of a handle (for example
+`context_synchronize`, `context_get_device`, `graph_node_set_params`). Their
+callers `HANDLE_RETURN` the value.
+
+### Context-scoped operations
+
+Operations that must run in a specific context use `invoke_in_context` /
+`invoke_in_context_or_undo` (propagating paths) and `cleanup_in_context`
+(deleters). They switch the current context, run the operation, and restore the
+caller's context. When restoration fails after the operation succeeded, the
+creation is undone and the restoration status is returned; the helper also
+records a thread-local detail (`take_last_error_detail()`) that the Cython error
+path appends to the raised `CUDAError`, so the user learns that the caller's
+context was not restored and which context is current. When both the operation
+and the restoration fail, the operation status is returned and the restoration
+failure is reported out of band. Tests inject restoration failures with
+`set_context_restore_fault_for_testing()`.
+
+### Reporting from non-propagating paths
+
+Deleters, CUDA callbacks and cleanup-after-failure cannot raise. They report
+through `report_cuda_error()` / `report_message()` (the `pw_*` wrappers
+decorate destroy calls with it), which emit a `cuda.core.CUDAWarning` through
+the Python warnings machinery when the interpreter is usable, deliver an
+escalated warning as an unraisable exception, and fall back to stderr when the
+GIL cannot be taken (for example during finalization). `CUDA_ERROR_DEINITIALIZED`
+is never reported because it means the driver is shutting down. No status is
+discarded silently anywhere in this layer, and nothing in this layer terminates
+the process; see `docs/source/error_handling.rst` and the "Failure handling"
+section of `AGENTS.md` for the policy.
+
 ## Usage from Cython
 
 ```cython