Skip to content

Commit c96bbef

Browse files
test(litellm): Require filelock>=4.0.3 on Python 3.10/3.11 (#8026)
### Description This fixes the `AI (3.10)` litellm flake. The test that fails changes from run to run. Recent example: `test_binary_content_encoding_mixed_content[False-False]`, which failed **7 times on master** between Sep 30 and Oct 13 ([job](https://github.com/getsentry/sentry-python/actions/runs/38237427670/job/114771446502)). Other failures were in `test_no_integration[False-False]` and `test_binary_content_encoding_image_url[True-False]`. All 10 litellm failures I looked at on master have the same cause: ``` /usr/lib/python3.10/threading.py:1011: in _bootstrap_inner _sys.settrace(_trace_hook) E RuntimeError: Cannot install a trace function while another trace function is being installed ``` **Root cause** - On CPython < 3.12, `_PyEval_SetTrace` uses a `static int reentrant` flag shared by the whole process. The flag stays set while audit hooks run ([python/cpython#98257](python/cpython#98257), fixed in 3.12). - `filelock` 3.30.0 through 4.0.2 registers a Python audit hook (`sys.addaudithook(_audit_fork_safety)`) when it is imported. The py3.10 litellm envs pin `filelock==3.32.4`, and `huggingface_hub` imports it during the first test. - pytest-cov calls `threading.settrace`, so every new thread calls `sys.settrace()` while it starts up. If the GIL switches while another thread is inside filelock's hook, the new thread's `settrace()` raises and the thread dies before it runs. - The tests create a new single-worker executor (`reset_litellm_executor`), and LiteLLM finishes the `gen_ai.chat` span in a success callback submitted to it. A dead worker means the callback never runs, so the test gets `len(chat_spans) == 0`. **Fix** `filelock` 4.0.3 fixed this upstream: it no longer registers the hook on CPython < 3.12 (`_SETTRACE_SURVIVES_AUDIT_HOOKS`). This PR adds a `py3.10,py3.11` rule, `filelock>=4.0.3`, to the litellm config, then regenerates `tox.ini` with `scripts/generate-test-files.sh --only litellm`. - Only the four `py3.10-litellm-*` blocks change; they now pin `filelock==4.1.0`. py3.12 and later are unaffected and unchanged. - Re-resolving those four blocks also moved other transitive pins, for example `anthropic` 1.0.0 → 1.13.0. - `releases.jsonl` is unchanged. `--only` would have pruned it, so I restored it. `package_dependencies.jsonl` only gains the 3 new resolution entries. No retries, sleeps, skipped tests or weaker assertions. **Reproduction:** getsentry/repro#60 It is a standalone script with no litellm and no pytest. On Python 3.10 with filelock 3.32.4, it lost 14 of 32,000 callbacks in one run, and 3 of 5 shorter runs (16,000 callbacks each) reproduced it. Under the same conditions it lost 0 of 32,000 with filelock 4.0.3, and 0 on Python 3.12. Original suite, run locally (`py3.10-litellm-v1.83.14`, with coverage): - With default settings: 0 of 20 runs failed. - With GIL switching set to happen more often (`sys.setswitchinterval(1e-6)` via a `-p` plugin; no test code changed): 4 of 20 runs hit the `RuntimeError`, and 3 of 20 failed. This is a **draft** until validation of the fix against both the repro and the original suite is posted here. Possible follow-up, not changed here: `py3.11-anthropic-v0.16.0` also pins `filelock==3.32.4`. #### Issues * resolves: #7014 * Linear: PY-2660 <!-- junior-request-attribution:start --> via Junior system actor `event`. <!-- junior-request-attribution:end --> <!-- junior-session-footer:start --> <!-- junior-conversation-id:agent-dispatch%3Adispatch_6df7374375b237816e130534b6439833 --> -- [View Junior Session](https://junior-prod.sentry.dev/conversations/agent-dispatch%3Adispatch_6df7374375b237816e130534b6439833) [[Sentry]](https://sentry.sentry.io/explore/conversations/agent-dispatch%3Adispatch_6df7374375b237816e130534b6439833/?project=4510944073809921) <!-- junior-session-footer:end --> Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com>
1 parent 2ffd383 commit c96bbef

3 files changed

Lines changed: 196 additions & 183 deletions

File tree

‎scripts/populate_tox/config.py‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -287,6 +287,15 @@
287287
"package": "litellm",
288288
"deps": {
289289
"*": ["anthropic", "google-genai", "pytest-asyncio", "httpx"],
290+
# filelock 3.30.0-4.0.2 registers a Python audit hook on import. On
291+
# CPython < 3.12, sys.settrace() is guarded by a process-wide flag
292+
# that stays set while audit hooks run, so with coverage enabled a
293+
# new thread can die in Thread._bootstrap with "Cannot install a
294+
# trace function while another trace function is being installed"
295+
# (python/cpython#98257). LiteLLM finishes spans in thread pool
296+
# callbacks, so the span is lost. filelock 4.0.3 skips the hook on
297+
# affected interpreters.
298+
"py3.10,py3.11": ["filelock>=4.0.3"],
290299
},
291300
"include": "<=1.96", # https://github.com/BerriAI/litellm/issues/36384
292301
},

0 commit comments

Comments
 (0)