coroutine/memory leak

[alexandershov]

Looks like coroutines are not getting garbage-collected when yappi is enabled. I've also encountered this behaviour in production and was able to "fix" it by profile_threads=False (with the obvious downsides that only main thread will be profiled) but in the below minimal repro profile_threads=False doesn't help.

Repro (coroutine count/rss stabilizes without yappi, but grows with it):

uv run repro.py --no-yappi
iter=   0  coroutines=   10001  exhausted=   10000  rss=31mb
iter=  10  coroutines=   10001  exhausted=   10000  rss=39mb
iter=  20  coroutines=   10001  exhausted=   10000  rss=39mb

uv run repro.py
iter=   0  coroutines=   10001  exhausted=   10000  rss=30mb
iter=  10  coroutines=  110001  exhausted=  110000  rss=55mb
iter=  20  coroutines=  210001  exhausted=  210000  rss=75mb

repro.py

#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.11"
# dependencies = ["yappi==1.6.10"]
# ///
"""Minimal repro: yappi leaks coroutine objects when profiling async code.

Run with profiling (leak):   uv run repro.py
Run without profiling (ok):  uv run repro.py --no-yappi
"""
import asyncio
import gc
import resource
import sys
import types

import yappi

PROFILE = "--no-yappi" not in sys.argv
COROS_PER_ITER = 10_000


async def noop():
    return


async def main():
    i = 0
    while True:
        if PROFILE:
            yappi.set_clock_type("cpu")
            yappi.start()
        await asyncio.gather(*(noop() for _ in range(COROS_PER_ITER)))
        yappi.stop()
        yappi.clear_stats()
        gc.collect()

        coroutines = sum(1 for o in gc.get_objects() if isinstance(o, types.CoroutineType))
        exhausted = sum(
            1
            for o in gc.get_objects()
            if isinstance(o, types.CoroutineType) and o.cr_frame is None
        )
        rss_mb = int(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / (1024 * 1024))
        if i % 10 == 0:
            print(
                f"iter={i:4d}  coroutines={coroutines:8d}  exhausted={exhausted:8d}  rss={rss_mb}mb",
                flush=True,
            )
        i += 1


if __name__ == "__main__":
    asyncio.run(main())

[sumerc]

Thanks for the report!

Fix is rdy: #218.

[sumerc]

Version 1.7.7 in release process containing this fix.