gh-140550: Update xxlimited with 3.15 limited API & PEP 697 #142827

encukou · 2025-12-16T14:21:03Z

Copy the existing xxlimited to xxlimited_3_13: this module serves as a rudimentary check that we don't break older Limited API. This is similar to xxlimited_35. (Nowadays we know that it's good to have a separator between the major/minor versions, but renaming to xxlimited_3_5 isn't worth the churn.)
Reorganize test_xxlimited to make similar copies easier in the future
Switch xxlimited to Limited API 3.15, PyModExport (PEP-793), standalone instance struct with extended basicsize (PEP-697)
treat PyObject as opaque, as proposed in PEP-803 and PEP-809, using a private knob (_Py_OPAQUE_PYOBJECT) for now to ensure we don't rely on PyObject layout details accidentally. (I'll replace/remove the private API in 3.15 beta at latest.)

Issue: Implement PEP 793 – PyModExport #140550

This adds the experimental free-threading support.

Lib/test/test_xxlimited.py

Modules/xxlimited.c

ZeroIntensity · 2025-12-16T16:04:18Z

Modules/xxlimited.c

-    XxoObject *self = XxoObject_CAST(op);
-    Py_VISIT(self->x_attr);
+    XxoObject_Data *data = Xxo_get_data(self);
+    if (data == NULL) {


Xxo_get_data will set an exception on failure, which will probably break the GC. Same for Xxo_clear below.

Yeah; the newly documented requirements for tp_traverse are breaking the design; I'll need to add some new borrowing+infallible APIs :/

Maybe call PyErr_WriteUnraisable()?

Same for Xxo_finalize(), no? Py_tp_finalize is not supposed to exit with an exception set. _Py_Dealloc() (tp_del) checks if an exception is set if Python is built in debug mode, but PyObject_CallFinalizer() (tp_finalize) doesn't.

PyErr_WriteUnraisable won't work, because tp_finalize can't create an exception object at all. It must not touch any refcounts; it's not enough to restore them to the previous state.

This is a problem. See: https://discuss.python.org/t/adding-c-api-for-use-in-tp-traverse/105331

ZeroIntensity · 2025-12-16T16:08:04Z

Modules/xxlimited.c

+    /* TODO: This is not quite true yet: there is a race in Xxo_setattro
+     * for example.
+     */


Wouldn't it be better to mark this as Py_MOD_GIL_USED in the meantime, or are you trying to avoid the warning?

AFAIK, the situation is unchanged since Py_MOD_GIL_NOT_USED was added everywhere in #116882.

I think the intention is/was to make everything thread-safe before some stage of free-threading implementation?

I think we added that everywhere under the assumption that everything was already thread-safe, and anything that should be reported to us as a bug. I'm not sure that applies with our own test suite.

This is not really a test suite, it supposed to be an example/template showing how to use the Limited API.
The limited API was never compatible with free-threading (yet), so I don't think there's a bug?

There's an earlier PR, #110764, which did this (Py_NOGIL was later renamed to Py_GIL_DISABLED):

+#ifndef Py_NOGIL #define Py_LIMITED_API 0x030d0000 +#endif

... but that seems to defeat the point of this module :/

Maybe we should have not built these modules at all in free-threading builds, and skip the relevant tests?
Doing that now seems like wasted effort though, when mutexes/critical sections are a genuine TODO item for the limited API.

If it's a template, I'm a little concerned that people will see this and think "oh, this module is thread-safe", when it's really not. The comment helps, but I don't know what percentage of people will ignore it.

encukou · 2025-12-16T17:57:52Z

I'll mark this draft -- this PR makes some shortcomings quite visible, and I guess it's better to address them before suggesting the new API :)

vstinner · 2025-12-17T12:21:26Z

Modules/xxlimited.c

-newXxoObject(PyObject *module)
+// Xxo initialization
+// This is the implementation of Xxo.__new__; it takes arbitrary positional
+// and keyword arguments.


I don't understand "it takes arbitrary positional and keyword arguments" since the error message below is "Xxo.__new__() takes no arguments".

The tp_new calling convention takes args&kwargs, so we need to check.
I agree the comment was misleading. Also the situation should be clear without the comment, so I removed it.

vstinner · 2025-12-17T12:23:15Z

Modules/xxlimited.c

-    XxoObject *self = XxoObject_CAST(op);
-    Py_VISIT(self->x_attr);
+    XxoObject_Data *data = Xxo_get_data(self);
+    if (data == NULL) {


Maybe call PyErr_WriteUnraisable()?

Same for Xxo_finalize(), no? Py_tp_finalize is not supposed to exit with an exception set. _Py_Dealloc() (tp_del) checks if an exception is set if Python is built in debug mode, but PyObject_CallFinalizer() (tp_finalize) doesn't.

vstinner · 2025-12-17T12:31:33Z

Modules/xxlimited.c

-    if (self->x_attr == NULL) {
+    // filter a specific attribute name
+    int is_reserved = PyUnicode_EqualToUTF8(name, "reserved");
+    if (is_reserved < 0) {


PyUnicode_EqualToUTF8() cannot return -1: https://docs.python.org/dev/c-api/unicode.html#c.PyUnicode_EqualToUTF8AndSize

encukou added 6 commits December 15, 2025 16:40

Copy xxlimited to xxlimited_3_13

8147b92

Run tests for all the xxlimited* modules

65f3fd0

Adjust module names in the 3_13 module

aa43538

Update xxlimited to 3.15 limited API (with opaque PyObject)

323695f

This adds the experimental free-threading support.

Comments & cleanups

597652b

Add Py_mod_abi

22464eb

encukou requested review from gpshead and vstinner December 16, 2025 14:21

encukou requested review from a team, AA-Turner, corona10, emmatyping, ericsnowcurrently and erlend-aasland as code owners December 16, 2025 14:21

bedevere-app bot added the awaiting core review label Dec 16, 2025

bedevere-app bot mentioned this pull request Dec 16, 2025

Implement PEP 793 – PyModExport #140550

Open

25 tasks

encukou mentioned this pull request Dec 16, 2025

gh-124153: Add Py_tp_token support to xxlimited module #141644

Draft

encukou added 2 commits December 16, 2025 15:33

Remove unused import

cc5e8f3

Don't flag static PyABIInfo variables

945b450

encukou added the skip news label Dec 16, 2025

encukou requested a review from ZeroIntensity December 16, 2025 14:44

erlend-aasland removed their request for review December 16, 2025 14:45

ZeroIntensity reviewed Dec 16, 2025

View reviewed changes

encukou marked this pull request as draft December 16, 2025 17:58

bedevere-app bot removed the awaiting core review label Dec 16, 2025

encukou added 3 commits December 17, 2025 12:02

Refcounting fix

3b3ae7c

Style

ff0f300

Expand TODO

29f1690

vstinner reviewed Dec 17, 2025

View reviewed changes

encukou added 2 commits December 17, 2025 14:56

Remove comment that might be misleading

d9728de

PyUnicode_EqualToUTF8 can't error (and doesn't type-check)

0bc0198

Uh oh!

gh-140550: Update xxlimited with 3.15 limited API & PEP 697 #142827

Are you sure you want to change the base?

gh-140550: Update xxlimited with 3.15 limited API & PEP 697 #142827

Uh oh!

Conversation

encukou commented Dec 16, 2025 • edited by bedevere-app bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

encukou Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

encukou Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

encukou commented Dec 16, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

encukou commented Dec 16, 2025 •

edited by bedevere-app bot

Loading

encukou Dec 17, 2025 •

edited

Loading

encukou Dec 16, 2025 •

edited

Loading