You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
PEP 788: Protecting the C API from Interpreter Finalization #70
I'd like the C API working group to begin considering PEP 788, "Protecting the C API from Interpreter Finalization", for recommendation to the SC.
The TL;DR is that we get a new PyThreadState_Ensure function to replace PyGILState_Ensure, which has a number of issues regarding subinterpreters and thread-safety. As part of introducing that, it includes several new interpreter "guarding" APIs that prevent an interpreter from shutting down, so things don't unexpectedly deallocate.
The main impact here is that the proposal deprecates PyGILState and schedules it for removal in a future version, so there will be a very huge migration from PyGILState_Ensure to PyThreadState_Ensure. APIs that use PyGILState_Ensure under the hood (for example, _PyObject_Dump in CPython) will probably need to be deprecated and given counterparts that take a correct PyInterpreterGuard parameter.
I'm not sure it's time to start formally voting on the proposal, but it would be good to hear some of the workgroup's feedback on the current design.
changed the title [-]PEP 788: PyInterpreterRef: Interpreter References[/-][+]PEP 788: PyInterpreterRef: Interpreter References in the C API[/+]on Jul 18, 2025
The int PyInterpreterRef_Get(PyInterpreterRef *ref) style functions might as well just return PyInterpreterRef, no? It's going to have to be an integer of some size (presumably pointer sized), so we can define 0/NULL as invalid, and simplify everyone's use of it.
A number of APIs "cannot fail" but they take a user-provided parameter, which means they can fail if given garbage input. What are they going to do in this case? Crash? Garbage out? Or better yet... fail 😉
Some also seem likely to involve memory allocation, unless you've got some real magic in mind for how to handle weak references (typically you need to allocate new memory for a weak reference to own, as well as adding it to a list owned by the live object so it can go through and mark them all as closed when it closes). Memory allocation can fail, which means functions like PyInterpreterWeakRef_Dup can fail (as well as probably needing to follow whatever rules we apply for allocator synchronisation and/or thread affinity). In short, "cannot fail" seems like a huge claim, and we're better off just saying that it could fail, even if failures are rare in practice.
I don't particularly like adding the dup model of reference counting at this stage, since we don't use it elsewhere (I know there's a proposal ongoing to bring it in, but that hasn't happened yet). How about PyInterpreterRef_NewRef (instead of _Dup) and PyInterpreterRef_NewWeakRef to get a weak reference?
AsStrong and AsInterpreter feel very unconventional. I'd have expected Get/GetTarget and GetInterpreterState here, to align with existing APIs that do similar things. Should also have a way to get a reference to a PyInterpreterState * (i.e. the opposite of AsInterpreter/GetInterpreterState).
Not a fan of PyInterpreter[Strong/Weak]Ref_Get for getting the current interpreter. We should only need to get a strong reference for the current interpreter (e.g. PyInterpreterRef_GetCurrent) and then can get a weak reference from that (NewWeakRef above).
What should I (as an embedder) be reporting if PyThreadState_Ensure fails? The example prints "out of memory", which seems presumptuous. Should I just be saying that some kind of internal Python error has occurred? "Failed to activate Python thread"?
PyInterpreterRef_Main - needs "get" somewhere in the name, and I think I'd prefer to see this as a PyRuntime_ function rather than PyInterpreterRef_. (We're also aiming to do away with the "main" interpreter as a concept, which means this kind of use will eventually be "an arbitrary default interpreter" rather than anything special. So an API to get "any"/"the first" active interpreter from the runtime is likely to align better with the intended design.)
(It'd probably be better to discuss these on the DPO post.)
The int PyInterpreterRef_Get(PyInterpreterRef *ref) style functions might as well just return PyInterpreterRef, no? It's going to have to be an integer of some size (presumably pointer sized), so we can define 0/NULL as invalid, and simplify everyone's use of it.
Maybe. I don't have too much of a preference. I originally designed it that way, but I think @vstinner preferred the current design.
A number of APIs "cannot fail" but they take a user-provided parameter, which means they can fail if given garbage input. What are they going to do in this case? Crash? Garbage out? Or better yet... fail 😉
It seems totally fine to me for these to crash/fatal error if you feed them garbage. They're just pointers (at least in the reference implementation), so if you play silly games, you'll get silly segfaults. How are you envisioning these failing?
Memory allocation can fail, which means functions like PyInterpreterWeakRef_Dup can fail (as well as probably needing to follow whatever rules we apply for allocator synchronisation and/or thread affinity). In short, "cannot fail" seems like a huge claim, and we're better off just saying that it could fail, even if failures are rare in practice.
Dup functions reuse the allocation and increment a reference count, but I think I agree that these should be failable to be more future-proof. I'll update the proposal once we come to a consensus on how things should fail.
I don't particularly like adding the dup model of reference counting at this stage, since we don't use it elsewhere (I know there's a proposal ongoing to bring it in, but that hasn't happened yet). How about PyInterpreterRef_NewRef (instead of _Dup) and PyInterpreterRef_NewWeakRef to get a weak reference?
PyInterpreterRef_NewRef sounds fine, but I really don't like PyInterpreterRef_NewWeakRef. That will make weak references especially annoying to acquire, because you'll have to acquire and release a strong reference just to get a weak reference. The boilerplate would be like this:
AsStrong and AsInterpreter feel very unconventional. I'd have expected Get/GetTarget and GetInterpreterState here, to align with existing APIs that do similar things.
GetTarget seems really unclear, but I'm fine with GetInterpreterState. I just chose AsInterpreter because it was shorter.
Should also have a way to get a reference to a PyInterpreterState * (i.e. the opposite of AsInterpreter/GetInterpreterState).
That defeats the whole point of having an interpreter reference in the first place; a PyInterpreterState * can be deallocated at any moment if you don't hold an attached thread state or strong ref for it, so it isn't thread-safe.
Not a fan of PyInterpreter[Strong/Weak]Ref_Get for getting the current interpreter. We should only need to get a strong reference for the current interpreter (e.g. PyInterpreterRef_GetCurrent) and then can get a weak reference from that (NewWeakRef above).
Yeah, ditto. Weak and strong references should definitely be separate here.
What should I (as an embedder) be reporting if PyThreadState_Ensure fails? The example prints "out of memory", which seems presumptuous. Should I just be saying that some kind of internal Python error has occurred? "Failed to activate Python thread"?
I think "out of memory" is a good fit. The idea is that PyThreadState_Ensure can only fail if creating the thread state fails, because the interpreter is guaranteed to be in a state where it can attach a thread state (because the caller holds a strong reference).
PyInterpreterRef_Main - needs "get" somewhere in the name, and I think I'd prefer to see this as a PyRuntime_ function rather than PyInterpreterRef_. (We're also aiming to do away with the "main" interpreter as a concept, which means this kind of use will eventually be "an arbitrary default interpreter" rather than anything special. So an API to get "any"/"the first" active interpreter from the runtime is likely to align better with the intended design.)
Yeah, this has been brought up a few times. My rationale for PyInterpreterRef_Main was to be the "PyInterpreterRef equivalent of PyInterpreterState_Main", but I can see how the name is confusing when that context is missing. I'll rename it to PyInterpreterState_GetMain.
I don't think we need to put this under PyRuntime (we don't have anything public in the PyRuntime namespace, do we? ) or try to mark it as an "arbitrary default interpreter". The whole point of PyInterpreterState_Main is to make an easier migration for PyGILState_Ensure users that really don't want to support subinterpreters. The only reason it's supposed to be thread-safe is because the main interpreter is statically allocated, so an arbitrary PyInterpreterState * wouldn't be a good fit.
Not sure at this stage, but factoring in other implementations that want to follow CPython's API, and the fact that we know 100% that we can't predict the future, allowing us to later make them fail seems like a better option than declaring it impossible. (Deprecation is one thing that's better when we can support failure, though it's always hard to distinguish between "failed and you should give up" and "fail and you should try a different approach" when we don't set exceptions.)
There's also a possibility that one day we might be able to set exceptions here, which makes failure worth supporting. Part of the reason for having a C API WG is for us to guide designs by future, unpredictable things that we've seen happen in the past, rather than always designing the minimal API for the current implementation (which is usually the reason that those "unpredictable" things caused such trouble).
GetTarget seems really unclear
Yeah, must be a different language that uses target in their API - I was thinking of PyWeakref_GetObject as the existing pattern. So Get or GetInterpreterRef would be better than AsStrong.
That defeats the whole point of having an interpreter reference in the first place; a PyInterpreterState * can be deallocated at any moment if you don't hold an attached thread state or strong ref for it, so it isn't thread-safe.
There are plenty of circumstances where you can reasonably assume that someone else (in your callstack) is holding a strong reference, but has just passed you the state pointer. There's plenty of existing code out there that doesn't use references (obviously), but may well deal in interpreter states. We make porting a lot easier by allowing to go both directions.
I think "out of memory" is a good fit.
Okay, then write that explicitly into the specification, not just in an example. If you can confidently say "this will only ever fail due to out of memory" then we can specify it (FWIW, I don't think you should confidently say that, and should go more generic).
My rationale for PyInterpreterRef_Main was to be the "PyInterpreterRef equivalent of PyInterpreterState_Main", but I can see how the name is confusing when that context is missing. I'll rename it to PyInterpreterState_GetMain.
Or perhaps PyInterpreterState_GetDefault[Ref] (I like it better with Ref since it's not returning a state pointer)? Setting the default can be defined as internal only at this stage, but it leaves us in a better future position to update the model without having to deprecate your APIs (e.g. if the "default" is later allowed to shut down before other interpreters, then a new interpreter can become default - this seems incredibly possible in an embedding situation).
(It'd probably be better to discuss these on the DPO post.)
Then why come here asking for feedback? If the idea has passed DPO discussion, then now we're specifically checking the API design. Everything I've mentioned is in scope for C API WG review (you'll notice I haven't criticised the somewhat rambling motivation or misuse of "rationale" for "rationalisations" 😉).
Not sure at this stage, but factoring in other implementations that want to follow CPython's API, and the fact that we know 100% that we can't predict the future, allowing us to later make them fail seems like a better option than declaring it impossible. (Deprecation is one thing that's better when we can support failure, though it's always hard to distinguish between "failed and you should give up" and "fail and you should try a different approach" when we don't set exceptions.)
Ok, that makes sense. My one concern is that some APIs (like PyInterpreterWeakRef_Dup) don't require you to hold an attached thread state, so Python can't really set an exception to denote what failed. I think that's probably work for a different proposal, though (maybe we could add a Py_errno for these cases)?
Yeah, must be a different language that uses target in their API - I was thinking of PyWeakref_GetObject as the existing pattern. So Get or GetInterpreterRef would be better than AsStrong.
I think that works for PyInterpreterRef_AsInterpreter (which I do see now is a little weirdly named in hindsight), but GetStrong is a little confusing to me, because in reality it's constructing something new rather than being a "getter". We could try a totally different name for it instead. Perhaps something like PyInterpreterWeakRef_Promote?
There are plenty of circumstances where you can reasonably assume that someone else (in your callstack) is holding a strong reference, but has just passed you the state pointer. There's plenty of existing code out there that doesn't use references (obviously), but may well deal in interpreter states. We make porting a lot easier by allowing to go both directions.
I see your point, but I'm really worried that people will use such an API to write unsafe code like this:
staticvoidblah(PyInterpreterState*interp)
{
// Fresh thread; no interpreter reference anywherePyInterpreterRefref=PyInterpreterState_AsRef(interp);
PyThreadState_Ensure(ref);
/* ... The illusion of thread safety ... */
}
But anyway, if there's a PyInterpreterState * pointer available, then it's reasonable to assume that PyInterpreterState_Get returns that same interpreter, right? If it is, no need for AsRef, and if not, then it's not thread safe (because the thread isn't attached to the interpreter and it may be deallocated at any moment).
(FWIW, I don't think you should confidently say that, and should go more generic).
Ok, I'll update the proposal to change the failure message in the examples. The current documentation for PyThreadState_Ensure in the PEP simply reads "Return 0 on success, and -1 without an exception set on failure." Is that broad enough for future-proofing?
I like it better with Ref since it's not returning a state pointer
Well, since there's no state pointer involved (other than internally), we probably should avoid the PyInterpreterState prefix. I'm okay with PyInterpreterRef_GetDefault, but ideally we'll get rid of that API someday anyway. People should be using references for a specific interpreter, not the arbitrary "default" one. It solely acts as a compatibility shim for those who use PyGILState_Ensure (which will be removed), but don't have the time to try to support subinterpreters.
Then why come here asking for feedback? If the idea has passed DPO discussion, then now we're specifically checking the API design. Everything I've mentioned is in scope for C API WG review (you'll notice I haven't criticised the somewhat rambling motivation or misuse of "rationale" for "rationalisations" 😉).
I meant just for visibility (primarily for the SC who'll be reading the threads). I wasn't at all trying to say that I didn't want the feedback, I greatly appreciate it!
I'm also fine if you criticize the text of the proposal, here or on DPO; I'd rather the WG be able to comfortably understand the PEP even if it means going out of "scope" to get there :)
In functions that don't need attached thread state, return -1 without raising an exception.
Deallocation functions (*_Close) are OK not failing.
The int PyInterpreterRef_Get(PyInterpreterRef *ref) style functions might as well just return PyInterpreterRef, no? It's going to have to be an integer of some size (presumably pointer sized), so we can define 0/NULL as invalid, and simplify everyone's use of it.
It's an opaque type, not an integer. You can't meaningfully compare it.
I don't particularly like adding the dup model of reference counting at this stage, since we don't use it elsewhere (I know there's a proposal ongoing to bring it in, but that hasn't happened yet). How about PyInterpreterRef_NewRef (instead of _Dup) and PyInterpreterRef_NewWeakRef to get a weak reference?
Well, if you're waiting for the proposal to bring it in, this is it. Regardless of what the operation is named.
To me personally, PyInterpreterRef_NewRef doesn't sounds right -- it should be _Copy, _Clone, or, well, _Dup.
AsStrong and AsInterpreter feel very unconventional. I'd have expected Get/GetTarget and GetInterpreterState here, to align with existing APIs that do similar things. Should also have a way to get a reference to a PyInterpreterState * (i.e. the opposite of AsInterpreter/GetInterpreterState).
Hm, that clashes with _Get for the constructor functions. Guess those should use _New or _From.
I am not familiar with this part of the C API, so my opinion is superficial. I do not want to block this this feature, so voted in favor of it, but I can't guarantee the absent of flaws in it.
Some minor notes and questions:
Py_IsFinalizing Is Not Atomic
The problem is not that Py_IsFinalizing() is not atomic, but that a sequence of checking Py_IsFinalizing() and attaching of a thread state is not atomic. This is just a minor inaccuracy.
Is PyGILState_Ensure() now deprecated? I hope that it can still be used if the code is not executed in a subinterpreter or non-Python daemon thread.
What happens if the PyInterpreterGuard or PyInterpreterView is never closed or the PyThreadView is never released? Will it prevent the finalization of the Python interpreter or there is a way to force close them?
PyThreadState_Ensure() returns a non-zero thread view of the old thread state on success, and 0 on failure. How to distinguish the case when there were no the old thread state from an error (when a new thread state cannot be created or an interpreter guard is not valid, etc)?
PyInterpreterView_Copy() on one hand "on failure, it returns 0 without an exception set", on other hand "This function cannot fail".
Exposing an Activate/Deactivate API Instead of Ensure/Clear
The current proposed version uses Ensure/Release, not Ensure/Clear.
Is PyGILState_Ensure() now deprecated? I hope that it can still be used if the code is not executed in a subinterpreter or non-Python daemon thread.
Yeah, I think it's still a good idea to deprecate PyGILState_Ensure even if threads aren't in use. It will still hang the thread if used after finalization (in the main thread) or if allocating memory fails. But, we should be working towards a C API that's better for subinterpreters anyway. If people really don't want to move their code away from PyGILState_Ensure, they can implement their own version using PyUnstable_InterpreterView_FromDefault.
What happens if the PyInterpreterGuard or PyInterpreterView is never closed or the PyThreadView is never released? Will it prevent the finalization of the Python interpreter or there is a way to force close them?
PyInterpreterView and PyThreadView will just cause a small memory leak when not released, but not releasing PyInterpreterGuard will cause a deadlock during finalization. I'll clarify this in the PEP.
PyThreadState_Ensure() returns a non-zero thread view of the old thread state on success, and 0 on failure. How to distinguish the case when there were no the old thread state from an error (when a new thread state cannot be created or an interpreter guard is not valid, etc)?
In the reference implementation, PyThreadState_Ensure returns a pointer to a global C sentinel to denote "no prior thread state attached", which is a bit hacky. I'm open to switching the signature to int PyThreadState_Ensure(PyThreadView *view) if this is a problem.
I'll address the rest of the comments with a small update to the PEP. Thanks for taking the time to review!
I'd like the C API working group to begin considering PEP 788, "Protecting the C API from Interpreter Finalization", for recommendation to the SC.
The C API Working Group approved your PEP 788. I hope that the back and forth helped to design a better API and clarify the PEP 😀 You can now submit your PEP to the SC. I close the issue.
The Steering Council is currently reviewing PEP 788. We appreciate the effort behind this proposal and agree that interpreter finalization can lead to subtle and difficult issues that deserve attention. Before we reach a decision, we have a number of questions that we believe would help us better evaluate the PEP.
From our reading so far, the proposed APIs appear relatively complex and potentially confusing for the average C API user, particularly in terms of their overall structure and terminology. We have reservations as to whether this level of complexity is justified, given that the issues the PEP aims to address seem to arise in relatively uncommon and niche scenarios. We are also considering whether it might be preferable to revisit the problem later with a more practical and intuitive approach, once there is broader experience or clearer demand for this kind of functionality.
With that in mind, we would like to ask the following questions.
How significant are subinterpreter related issues in practice, and are extension authors actively encountering them today?
Is lock acquisition during interpreter shutdown a frequent enough problem to warrant dedicated API support?
What is the level of support for PEP 788 within the C API Working Group? We would welcome the C API WG’s detailed perspective on this PEP.
Are there any end-to-end, real world examples of extension modules that would meaningfully benefit from these APIs that you can share
Warm regards from very, very cold Seoul,
Donghee
on behalf of the Python Steering Council
I think some of these questions are due to the motivation being poorly constructed. Would it help the SC's decision if I did an editorial pass to make it clearer?
We are also considering whether it might be preferable to revisit the problem later with a more practical and intuitive approach, once there is broader experience or clearer demand for this kind of functionality.
For what it's worth, PEP 788's approach is not novel. A similar design using special atexit handlers has been the norm for this in the past.
PEP 788 does propose some new concepts, but the fundamental mechanism for preventing finalization isn't a completely new concept. I should definitely clarify this in the PEP, sorry!
How significant are subinterpreter related issues in practice, and are extension authors actively encountering them today?
They're not very common, primarily because subinterpreters currently lack widespread usage. But, I think this will probably change with the recent introduction of concurrent.interpreters.
More generally though, the subinterpreter issues coincide with existing problems regarding usage of the C API during finalization; they're just more extreme, because the interpreter state allocation is freed in addition to clearing it. (This might happen for the main interpreter too some day -- we don't want to limit ourselves to keeping the main interpreter statically allocated.)
I think it would be very difficult to design an API that fixes the main interpreter but does not improve the situation for subinterpreters, because in both cases, we just want a mechanism to temporarily prevent finalization.
Is lock acquisition during interpreter shutdown a frequent enough problem to warrant dedicated API support?
This one does happen and poses an annoying bug for users. The PEP links python/cpython#129536 as an example. This has also come up several times on StackOverflow.
To be clear, PEP 788 does not fix usage of Python locks during finalization, because those are still going to be prone to re-entrancy deadlocks. The proposed API is only helpful for locks that are acquired in non-re-entrant C code, such as the stdout/stderr locks.
What is the level of support for PEP 788 within the C API Working Group? We would welcome the C API WG’s detailed perspective on this PEP.
I'll try not to speak for anyone, but I didn't hear any major complaints from the members of the C API WG that attended the core sprint in September.
That said, @ericsnowcurrently did share the SC's concern that the proposed API introduced some complexity. If I understood his POV correctly, I think he wanted something similar to the initial revision of this proposal, where a guard was implicit:
This is a similar design to the initial revision of this proposal, which was not received well. TL;DR making an arbitrary PyInterpreterState * implicitly guarded or non-guarded is very confusing.
Are there any end-to-end, real world examples of extension modules that would meaningfully benefit from these APIs that you can share
We would welcome the C API WG’s detailed perspective on this PEP.
The perspective of the WG as a whole was incorporated into the PEP. The discussions were quite long on this one :)
I'm sure there are details each of us would have done differently, of course, but this the best proposal we collectively can come up with now.
We are also considering whether it might be preferable to revisit the problem later with a more practical and intuitive approach, once there is broader experience or clearer demand for this kind of functionality.
IMO, that would mean adding these provisionally as PyUnstable, and delaying the deprecations. I don't think we can get broader experience with the current API.
Reacted by Steve Dower, Peter Bierma and Donghee Na
The Steering Council is generally positive about the newly revised PEP 788 and expects to approve it, unless there are major objections from the C API WG or from the binding ecosystem perspective.
If no major objections are raised by AoE on April 27, 2026, we plan to formally approve the PEP next week.
If the C API WG or members of the binding ecosystem have any major objections or concerns,
please either raise them in the thread or send them to steering-council@python.org.
We will review any such feedback before the final official approval.
Warm regards,
Donghee
on behalf of the Python Steering Council
PyO3 doesn't yet support multiple interpreters, so that part of the proposal doesn't impact us. That said, I'm sure the people working on subinterpreters would appreciate having APIs that are safer to use under multiple interpreters.
I'm also very happy to use the new API for attaching and detaching. PyO3 recently completed a migration to rename the old allow_threads and with_gil function to detach and attach and this new API nicely matches that naming we have in PyO3 now.
I wrote a whole section of the porting guide in the free-threaded Python guide that tries to make it clear that it's still necessary to explicitly attach and detach from the interpreter on the free-threaded build.
By far the most common source of confusion when people port old C extensions to support the free-threaded interpreter is misconceptions around the need to keep code that uses PyGILState_Ensure and PyGILState_Release, despite the lack of a GIL on the free-threaded build. Moving away from the old terminology will go a long way towards easing that confusion. The fact that the new API also fixes a longstanding issue with subinterpreters makes it even more obvious that this is a good idea.
SC have decided to move the veto timeline announced in the original post up from May 1 to April 27 AoE.
This reflects internal discussion that, if the PEP is officially approved, the implementer should have enough time to proceed with the implementation before the beta cutoff (May 5).
I also want to apologize for any confusion caused, especially since I was the one who made the original post.
Warm regards,
Donghee
on behalf of the Python Steering Council
Activity
vstinner commentedon Jun 30, 2025
I'm the sponsor of this PEP and I support it :-) It solves an old big design issue with C threads leading to crashes or hangs.
encukou commentedon Jun 30, 2025
I'll post the discussion thread, but yes, I'm happy with the PEP overall. Thank you!
vstinner commentedon Jun 30, 2025
Link to the discussion: https://discuss.python.org/t/pep-788-reimagining-native-threads-new-updates/93653
[-]PEP 788: Reimagining Native Threads[/-][+]PEP 788: Interpreter References[/+][-]PEP 788: Interpreter References[/-][+]PEP 788: PyInterpreterRef: Interpreter References[/+]encukou commentedon Jul 16, 2025
Looks like the discussion died down; let's vote!
[-]PEP 788: PyInterpreterRef: Interpreter References[/-][+]PEP 788: PyInterpreterRef: Interpreter References in the C API[/+]ZeroIntensity commentedon Jul 27, 2025
Friendly bump
encukou commentedon Jul 30, 2025
@erlend-aasland @mdboom @serhiy-storchaka @zooba, what are your thoughts on the PEP?
zooba commentedon Jul 30, 2025
The
int PyInterpreterRef_Get(PyInterpreterRef *ref)style functions might as well just returnPyInterpreterRef, no? It's going to have to be an integer of some size (presumably pointer sized), so we can define0/NULLas invalid, and simplify everyone's use of it.A number of APIs "cannot fail" but they take a user-provided parameter, which means they can fail if given garbage input. What are they going to do in this case? Crash? Garbage out? Or better yet... fail 😉
Some also seem likely to involve memory allocation, unless you've got some real magic in mind for how to handle weak references (typically you need to allocate new memory for a weak reference to own, as well as adding it to a list owned by the live object so it can go through and mark them all as closed when it closes). Memory allocation can fail, which means functions like
PyInterpreterWeakRef_Dupcan fail (as well as probably needing to follow whatever rules we apply for allocator synchronisation and/or thread affinity). In short, "cannot fail" seems like a huge claim, and we're better off just saying that it could fail, even if failures are rare in practice.I don't particularly like adding the
dupmodel of reference counting at this stage, since we don't use it elsewhere (I know there's a proposal ongoing to bring it in, but that hasn't happened yet). How aboutPyInterpreterRef_NewRef(instead of_Dup) andPyInterpreterRef_NewWeakRefto get a weak reference?AsStrongandAsInterpreterfeel very unconventional. I'd have expectedGet/GetTargetandGetInterpreterStatehere, to align with existing APIs that do similar things. Should also have a way to get a reference to aPyInterpreterState *(i.e. the opposite ofAsInterpreter/GetInterpreterState).Not a fan of
PyInterpreter[Strong/Weak]Ref_Getfor getting the current interpreter. We should only need to get a strong reference for the current interpreter (e.g.PyInterpreterRef_GetCurrent) and then can get a weak reference from that (NewWeakRefabove).What should I (as an embedder) be reporting if
PyThreadState_Ensurefails? The example prints "out of memory", which seems presumptuous. Should I just be saying that some kind of internal Python error has occurred? "Failed to activate Python thread"?PyInterpreterRef_Main- needs "get" somewhere in the name, and I think I'd prefer to see this as aPyRuntime_function rather thanPyInterpreterRef_. (We're also aiming to do away with the "main" interpreter as a concept, which means this kind of use will eventually be "an arbitrary default interpreter" rather than anything special. So an API to get "any"/"the first" active interpreter from the runtime is likely to align better with the intended design.)ZeroIntensity commentedon Jul 30, 2025
(It'd probably be better to discuss these on the DPO post.)
Maybe. I don't have too much of a preference. I originally designed it that way, but I think @vstinner preferred the current design.
It seems totally fine to me for these to crash/fatal error if you feed them garbage. They're just pointers (at least in the reference implementation), so if you play silly games, you'll get silly segfaults. How are you envisioning these failing?
Dupfunctions reuse the allocation and increment a reference count, but I think I agree that these should be failable to be more future-proof. I'll update the proposal once we come to a consensus on how things should fail.PyInterpreterRef_NewRefsounds fine, but I really don't likePyInterpreterRef_NewWeakRef. That will make weak references especially annoying to acquire, because you'll have to acquire and release a strong reference just to get a weak reference. The boilerplate would be like this:GetTargetseems really unclear, but I'm fine withGetInterpreterState. I just choseAsInterpreterbecause it was shorter.That defeats the whole point of having an interpreter reference in the first place; a
PyInterpreterState *can be deallocated at any moment if you don't hold an attached thread state or strong ref for it, so it isn't thread-safe.Yeah, ditto. Weak and strong references should definitely be separate here.
I think "out of memory" is a good fit. The idea is that
PyThreadState_Ensurecan only fail if creating the thread state fails, because the interpreter is guaranteed to be in a state where it can attach a thread state (because the caller holds a strong reference).Yeah, this has been brought up a few times. My rationale for
PyInterpreterRef_Mainwas to be the "PyInterpreterRefequivalent ofPyInterpreterState_Main", but I can see how the name is confusing when that context is missing. I'll rename it toPyInterpreterState_GetMain.I don't think we need to put this under
PyRuntime(we don't have anything public in thePyRuntimenamespace, do we? ) or try to mark it as an "arbitrary default interpreter". The whole point ofPyInterpreterState_Mainis to make an easier migration forPyGILState_Ensureusers that really don't want to support subinterpreters. The only reason it's supposed to be thread-safe is because the main interpreter is statically allocated, so an arbitraryPyInterpreterState *wouldn't be a good fit.zooba commentedon Jul 30, 2025
Not sure at this stage, but factoring in other implementations that want to follow CPython's API, and the fact that we know 100% that we can't predict the future, allowing us to later make them fail seems like a better option than declaring it impossible. (Deprecation is one thing that's better when we can support failure, though it's always hard to distinguish between "failed and you should give up" and "fail and you should try a different approach" when we don't set exceptions.)
There's also a possibility that one day we might be able to set exceptions here, which makes failure worth supporting. Part of the reason for having a C API WG is for us to guide designs by future, unpredictable things that we've seen happen in the past, rather than always designing the minimal API for the current implementation (which is usually the reason that those "unpredictable" things caused such trouble).
Yeah, must be a different language that uses target in their API - I was thinking of
PyWeakref_GetObjectas the existing pattern. SoGetorGetInterpreterRefwould be better thanAsStrong.There are plenty of circumstances where you can reasonably assume that someone else (in your callstack) is holding a strong reference, but has just passed you the state pointer. There's plenty of existing code out there that doesn't use references (obviously), but may well deal in interpreter states. We make porting a lot easier by allowing to go both directions.
Okay, then write that explicitly into the specification, not just in an example. If you can confidently say "this will only ever fail due to out of memory" then we can specify it (FWIW, I don't think you should confidently say that, and should go more generic).
Or perhaps
PyInterpreterState_GetDefault[Ref](I like it better withRefsince it's not returning a state pointer)? Setting the default can be defined as internal only at this stage, but it leaves us in a better future position to update the model without having to deprecate your APIs (e.g. if the "default" is later allowed to shut down before other interpreters, then a new interpreter can become default - this seems incredibly possible in an embedding situation).Then why come here asking for feedback? If the idea has passed DPO discussion, then now we're specifically checking the API design. Everything I've mentioned is in scope for C API WG review (you'll notice I haven't criticised the somewhat rambling motivation or misuse of "rationale" for "rationalisations" 😉).
ZeroIntensity commentedon Jul 31, 2025
Ok, that makes sense. My one concern is that some APIs (like
PyInterpreterWeakRef_Dup) don't require you to hold an attached thread state, so Python can't really set an exception to denote what failed. I think that's probably work for a different proposal, though (maybe we could add aPy_errnofor these cases)?I think that works for
PyInterpreterRef_AsInterpreter(which I do see now is a little weirdly named in hindsight), butGetStrongis a little confusing to me, because in reality it's constructing something new rather than being a "getter". We could try a totally different name for it instead. Perhaps something likePyInterpreterWeakRef_Promote?I see your point, but I'm really worried that people will use such an API to write unsafe code like this:
But anyway, if there's a
PyInterpreterState *pointer available, then it's reasonable to assume thatPyInterpreterState_Getreturns that same interpreter, right? If it is, no need forAsRef, and if not, then it's not thread safe (because the thread isn't attached to the interpreter and it may be deallocated at any moment).Ok, I'll update the proposal to change the failure message in the examples. The current documentation for
PyThreadState_Ensurein the PEP simply reads "Return 0 on success, and -1 without an exception set on failure." Is that broad enough for future-proofing?Well, since there's no state pointer involved (other than internally), we probably should avoid the
PyInterpreterStateprefix. I'm okay withPyInterpreterRef_GetDefault, but ideally we'll get rid of that API someday anyway. People should be using references for a specific interpreter, not the arbitrary "default" one. It solely acts as a compatibility shim for those who usePyGILState_Ensure(which will be removed), but don't have the time to try to support subinterpreters.I meant just for visibility (primarily for the SC who'll be reading the threads). I wasn't at all trying to say that I didn't want the feedback, I greatly appreciate it!
I'm also fine if you criticize the text of the proposal, here or on DPO; I'd rather the WG be able to comfortably understand the PEP even if it means going out of "scope" to get there :)
encukou commentedon Aug 1, 2025
Good point about the fallible API.
In functions that don't need attached thread state, return -1 without raising an exception.
Deallocation functions (
*_Close) are OK not failing.It's an opaque type, not an integer. You can't meaningfully compare it.
Well, if you're waiting for the proposal to bring it in, this is it. Regardless of what the operation is named.
To me personally,
PyInterpreterRef_NewRefdoesn't sounds right -- it should be_Copy,_Clone, or, well,_Dup.Hm, that clashes with
_Getfor the constructor functions. Guess those should use_Newor_From.29 remaining items
serhiy-storchaka commentedon Oct 28, 2025
I am not familiar with this part of the C API, so my opinion is superficial. I do not want to block this this feature, so voted in favor of it, but I can't guarantee the absent of flaws in it.
Some minor notes and questions:
The problem is not that
Py_IsFinalizing()is not atomic, but that a sequence of checkingPy_IsFinalizing()and attaching of a thread state is not atomic. This is just a minor inaccuracy.Is
PyGILState_Ensure()now deprecated? I hope that it can still be used if the code is not executed in a subinterpreter or non-Python daemon thread.What happens if the
PyInterpreterGuardorPyInterpreterViewis never closed or thePyThreadViewis never released? Will it prevent the finalization of the Python interpreter or there is a way to force close them?PyThreadState_Ensure()returns a non-zero thread view of the old thread state on success, and 0 on failure. How to distinguish the case when there were no the old thread state from an error (when a new thread state cannot be created or an interpreter guard is not valid, etc)?PyInterpreterView_Copy()on one hand "on failure, it returns 0 without an exception set", on other hand "This function cannot fail".The current proposed version uses Ensure/Release, not Ensure/Clear.
ZeroIntensity commentedon Oct 28, 2025
Yeah, I think it's still a good idea to deprecate
PyGILState_Ensureeven if threads aren't in use. It will still hang the thread if used after finalization (in the main thread) or if allocating memory fails. But, we should be working towards a C API that's better for subinterpreters anyway. If people really don't want to move their code away fromPyGILState_Ensure, they can implement their own version usingPyUnstable_InterpreterView_FromDefault.PyInterpreterViewandPyThreadViewwill just cause a small memory leak when not released, but not releasingPyInterpreterGuardwill cause a deadlock during finalization. I'll clarify this in the PEP.In the reference implementation,
PyThreadState_Ensurereturns a pointer to a global C sentinel to denote "no prior thread state attached", which is a bit hacky. I'm open to switching the signature toint PyThreadState_Ensure(PyThreadView *view)if this is a problem.I'll address the rest of the comments with a small update to the PEP. Thanks for taking the time to review!
vstinner commentedon Oct 28, 2025
@ZeroIntensity updated the PEP: python/peps#4678
@ZeroIntensity:
The C API Working Group approved your PEP 788. I hope that the back and forth helped to design a better API and clarify the PEP 😀 You can now submit your PEP to the SC. I close the issue.
ZeroIntensity commentedon Oct 28, 2025
The PEP has been submitted: python/steering-council#316
corona10 commentedon Feb 4, 2026
The Steering Council is currently reviewing PEP 788. We appreciate the effort behind this proposal and agree that interpreter finalization can lead to subtle and difficult issues that deserve attention. Before we reach a decision, we have a number of questions that we believe would help us better evaluate the PEP.
From our reading so far, the proposed APIs appear relatively complex and potentially confusing for the average C API user, particularly in terms of their overall structure and terminology. We have reservations as to whether this level of complexity is justified, given that the issues the PEP aims to address seem to arise in relatively uncommon and niche scenarios. We are also considering whether it might be preferable to revisit the problem later with a more practical and intuitive approach, once there is broader experience or clearer demand for this kind of functionality.
With that in mind, we would like to ask the following questions.
Warm regards from very, very cold Seoul,
Donghee
on behalf of the Python Steering Council
ZeroIntensity commentedon Feb 4, 2026
Hi Donghee, thanks for the feedback.
I think some of these questions are due to the motivation being poorly constructed. Would it help the SC's decision if I did an editorial pass to make it clearer?
For what it's worth, PEP 788's approach is not novel. A similar design using special
atexithandlers has been the norm for this in the past.Here are some examples, courtesy of Matt:
PEP 788 does propose some new concepts, but the fundamental mechanism for preventing finalization isn't a completely new concept. I should definitely clarify this in the PEP, sorry!
They're not very common, primarily because subinterpreters currently lack widespread usage. But, I think this will probably change with the recent introduction of
concurrent.interpreters.More generally though, the subinterpreter issues coincide with existing problems regarding usage of the C API during finalization; they're just more extreme, because the interpreter state allocation is freed in addition to clearing it. (This might happen for the main interpreter too some day -- we don't want to limit ourselves to keeping the main interpreter statically allocated.)
I think it would be very difficult to design an API that fixes the main interpreter but does not improve the situation for subinterpreters, because in both cases, we just want a mechanism to temporarily prevent finalization.
This one does happen and poses an annoying bug for users. The PEP links python/cpython#129536 as an example. This has also come up several times on StackOverflow.
To be clear, PEP 788 does not fix usage of Python locks during finalization, because those are still going to be prone to re-entrancy deadlocks. The proposed API is only helpful for locks that are acquired in non-re-entrant C code, such as the
stdout/stderrlocks.I'll try not to speak for anyone, but I didn't hear any major complaints from the members of the C API WG that attended the core sprint in September.
That said, @ericsnowcurrently did share the SC's concern that the proposed API introduced some complexity. If I understood his POV correctly, I think he wanted something similar to the initial revision of this proposal, where a guard was implicit:
This is a similar design to the initial revision of this proposal, which was not received well. TL;DR making an arbitrary
PyInterpreterState *implicitly guarded or non-guarded is very confusing.PyArrow, according to Antoine, and Nanobind/pybind11, as they originally requested the API.
encukou commentedon Feb 9, 2026
The perspective of the WG as a whole was incorporated into the PEP. The discussions were quite long on this one :)
I'm sure there are details each of us would have done differently, of course, but this the best proposal we collectively can come up with now.
IMO, that would mean adding these provisionally as
PyUnstable, and delaying the deprecations. I don't think we can get broader experience with the current API.corona10 commentedon Apr 18, 2026
@ZeroIntensity @vstinner @encukou @erlend-aasland @zooba @serhiy-storchaka
The Steering Council is generally positive about the newly revised PEP 788 and expects to approve it, unless there are major objections from the C API WG or from the binding ecosystem perspective.
If no major objections are raised by AoE on April 27, 2026, we plan to formally approve the PEP next week.
Revised records
Warm regards,
Donghee
on behalf of the Python Steering Council
(2026-04-18) The veto timeline has been updated: #70 (comment)
corona10 commentedon Apr 18, 2026
@vstinner @encukou @erlend-aasland @zooba @serhiy-storchaka (as C API WG)
@ngoldbaum @wjakob (as binding ecosystems)
If the C API WG or members of the binding ecosystem have any major objections or concerns,
please either raise them in the thread or send them to steering-council@python.org.
We will review any such feedback before the final official approval.
Warm regards,
Donghee
on behalf of the Python Steering Council
ngoldbaum commentedon Apr 18, 2026
PyO3 doesn't yet support multiple interpreters, so that part of the proposal doesn't impact us. That said, I'm sure the people working on subinterpreters would appreciate having APIs that are safer to use under multiple interpreters.
I'm also very happy to use the new API for attaching and detaching. PyO3 recently completed a migration to rename the old
allow_threadsandwith_gilfunction todetachandattachand this new API nicely matches that naming we have in PyO3 now.I wrote a whole section of the porting guide in the free-threaded Python guide that tries to make it clear that it's still necessary to explicitly attach and detach from the interpreter on the free-threaded build.
By far the most common source of confusion when people port old C extensions to support the free-threaded interpreter is misconceptions around the need to keep code that uses
PyGILState_EnsureandPyGILState_Release, despite the lack of a GIL on the free-threaded build. Moving away from the old terminology will go a long way towards easing that confusion. The fact that the new API also fixes a longstanding issue with subinterpreters makes it even more obvious that this is a good idea.corona10 commentedon Apr 18, 2026
Hi,
SC have decided to move the veto timeline announced in the original post up from May 1 to April 27 AoE.
This reflects internal discussion that, if the PEP is officially approved, the implementer should have enough time to proceed with the implementation before the beta cutoff (May 5).
I also want to apologize for any confusion caused, especially since I was the one who made the original post.
Warm regards,
Donghee
on behalf of the Python Steering Council
cc @vstinner @encukou @erlend-aasland @zooba @serhiy-storchaka @ngoldbaum @wjakob
wjakob commentedon Apr 19, 2026
I am very happy with this proposal. It would fully address the concerns I had raised here. The API looks easy to integrate into nanobind.
encukou commentedon Apr 23, 2026
+1 for this version of the PEP.
My objections are adequately covered in Rejected ideas :)
davidhewitt commentedon Apr 23, 2026
Agreed with @ngoldbaum that this looks great for PyO3's purposes, thank you.