Do not use mimalloc on macOS (prevent pyarrow memory pool corruption)

[alexeyshockov]

Hope you don't mind an AI-assisted PR, since the change is small and focused on a specific bug.

The problem

pyarrow statically links its own copy of mimalloc and uses it as the default memory pool on macOS. pyreqwest links a second static copy as its #[global_allocator]. On macOS mimalloc locates its per-thread heap through hardcoded pthread TSD slots, so the two copies collide: whichever initializes last claims the slot, and the other then reads a foreign heap as its own.

The result is silent memory corruption in pyarrow. No network call, no pyreqwest API use — the import is enough:

import pyarrow as pa
import pyreqwest.exceptions
pa.table({"sales": [10, 20]}).to_pydict()
# {'sales': [0, 4527375348]}

Tables allocated before the import survive; everything allocated after is garbage. Order matters — import pyreqwest before import pyarrow is fine, which is what makes this so easy to hit by accident and so confusing to debug.

Verified on macOS arm64, Python 3.12 and 3.13, pyarrow 25.0.0, pyreqwest 0.12.2 (published wheel), in a clean venv containing only those two packages.

Two things confirm the diagnosis rather than merely correlate with it:

• ARROW_DEFAULT_MEMORY_POOL=system makes it go away — removing pyarrow's mimalloc copy resolves it.
• Building pyreqwest 0.12.2 without --features mimalloc makes it go away. I built both variants from the v0.12.2 tag to check.

It is specifically two mimallocs, not "a Rust extension loaded after pyarrow": polars is a Rust extension that loads after pyarrow without trouble, and it has no mimalloc in it — it uses the system allocator on macOS arm64.

This is arguably upstream mimalloc's to make impossible (there is a "fix for multiple mimalloc instances in one executable" as far back as v1.6.2, and the macOS TLS slot numbers have churned 108/109 → 126/127 → back to 108/109 in 3.4.3), but that has been a long-running battle and pyarrow's macOS default is not going to move.

The change

Gate the dependency on the target, not just the #[global_allocator] item, so the second copy is never linked into a macOS build at all. Everything else keeps mimalloc exactly as today — on other platforms mimalloc uses ordinary per-DSO TLS, where two copies do not share state.

cargo tree --features mimalloc on macOS resolves without mimalloc; --target x86_64-unknown-linux-gnu --features mimalloc still pulls in mimalloc v0.1.52. CI can keep passing --features mimalloc for macOS targets unchanged — it simply becomes a no-op there.

I have not verified the Linux side empirically (no Linux box here), only that the dependency still resolves for that target. If pyarrow turns out to default to mimalloc rather than jemalloc on some Linux build, the same gate would be wanted there too.

What it costs

I benchmarked both builds with this repo's own harness — tests/bench/latency.py, pyreqwest_st, granian + TLS echo server — alternating variants across two passes on macOS arm64 / Python 3.12, to keep thermal drift from favoring either side.

Body	Concurrency	with mimalloc	without	cost
10 KB	2	10.70 ms	11.03 ms	+3.0%
10 KB	100	7.56 ms	7.74 ms	+2.4%
10 KB	10	8.47 ms	8.66 ms	+2.2%
100 KB	2	23.58 ms	24.00 ms	+1.7%
100 KB	100	14.32 ms	14.53 ms	+1.5%
1 MB	2	132.7 ms	134.3 ms	+1.2%
1 MB	10	88.8 ms	89.8 ms	+1.0%
1 MB	100	72.3 ms	72.6 ms	+0.4%
2 MB	100	232.9 ms	234.5 ms	+0.7%
2 MB	10	229.5 ms	230.1 ms	+0.2%
2 MB	2	305.3 ms	305.2 ms	−0.1%

So roughly 2–3% on small responses, ~1% at 1 MB, ~0% at 2 MB, on macOS only. The gradient is monotone in body size, which is what you would expect if the win is per-allocation overhead amortized against body copying. One cell (100 KB / concurrency 10) produced a 19.6 ms outlier against its own 16.9 ms twin in the other pass; I have left it out rather than let noise claim a regression. mimalloc won 10 of the remaining 11 cells. Wheel size differs by 52 KB.

That is a real cost and it is your call whether it is worth paying. Alternatives if you would rather not: publish a separate no-mimalloc wheel variant for macOS, or document the incompatibility so people can build from source without the feature. Happy to rework this in either direction.

Checks

• pytest — 734 passed, 62 snapshots
• cargo fmt --check, cargo clippy --features mimalloc -- -D warnings — clean
• macOS release wheel built with --features mimalloc: no mimalloc in the binary, pyarrow uninjured, requests work

Context

Found while adding a pyreqwest transport to typicall, which also offers an Arrow/dataframe converter — so the two extras land in one process routinely.

🤖 Generated with Claude Code

[MarkusSintonen]

Thanks, pretty interesting! Do you find any relevant issue from mimalloc side?

[alexeyshockov]

This is a known mimalloc bug, already fixed upstream — twice — but neither fix can reach pyreqwest yet. That makes this PR a stopgap with a clear exit condition rather than a permanent removal.

The bug. microsoft/mimalloc#1327 is nearly our setup: a PyO3 extension with mimalloc as its #[global_allocator], co-loaded with pyarrow. Mechanism: mimalloc >= 3.2.6 keeps the per-thread heap pointer in fixed pthread TCB slots 108/109, which are process-global per thread — so co-resident instances walk each other's mi_theap_t* with their own struct offsets. Symbol visibility does not help; Arrow already marks its mimalloc symbols private. It is a v2 -> v3 regression (#1301): v2 gated the fixed slot behind MI_MALLOC_OVERRIDE.

The fixes. mimalloc v3.4.4 (2026-08-01) switched macOS to pthread TLS by default — maintainer comment. Arrow did the same in apache/arrow#50549, merged after 25.0.0 branched, so PyPI pyarrow is still exposed.

Why we cannot use either. libmimalloc-sys 0.1.49 is the newest release and vendors mimalloc 3.3.02 by default — inside the >= 3.2.6 window, predating both MI_APPLE_TLS_THREAD_LOCAL and the 3.4.4 default — and its build.rs exposes no MI_TLS_MODEL_* knob. (local_dynamic_tls, which we enable, only sets -ftls-model=local-dynamic on the C compile: ELF-only, a no-op for macOS TSD slots.)

So the macOS choice today is binary: mimalloc, or pyarrow's memory intact. The gate can come out once libmimalloc-sys ships >= 3.4.4 — happy to say so in Cargo.toml. And the ~2-3% is leaving anyway: 3.4.4 gives up the same fast path upstream.

[MarkusSintonen]

So is it so that mimalloc has fixed it but not the rust crate yet? Is the fix landing soon to rust crate also?

[alexeyshockov]

Yes, mimalloc "fixed" it (they changed changed the default behaviour) in 3.4.4+ There is no open PRs for the Rust crate to upgrade, so I have no idea about their plans.

[MarkusSintonen]

Yes, mimalloc "fixed" it (they changed changed the default behaviour) in 3.4.4+ There is no open PRs for the Rust crate to upgrade, so I have no idea about their plans.

Possible if you could open issue on their end also?

[alexeyshockov]

You are right of course, this is the proper way to do that :) I just wanted to get the lib working, and fixing the whole chain is definitely not a fast path to it.

P.S.

Any reason to use #[global_allocator] in general? I'm not an expert is Rust, but AFAIK for a Python extension it's something that shouldn't be used by default, only after a justification. I quickly went through the code/git history and haven't really found one. My local benchmarks also does not show any meaningful gains from it, like all below 5% (usually much less).

IMO it will make the lib more reliable by default, decreasing chance of any cross-extensions bugs (like in the PR description).

[MarkusSintonen]

There is PR pending in rust mimalloc but it has not been merged yet purpleprotocol/mimalloc_rust#169

So Ill merge this and release in the meanwhile. Thank you for investigating!