Repository navigation
Commit c96bbef
authored
test(litellm): Require filelock>=4.0.3 on Python 3.10/3.11 (#8026)
### Description
This fixes the `AI (3.10)` litellm flake. The test that fails changes
from run to run. Recent example:
`test_binary_content_encoding_mixed_content[False-False]`, which failed
**7 times on master** between Sep 30 and Oct 13
([job](https://github.com/getsentry/sentry-python/actions/runs/38237427670/job/114771446502)).
Other failures were in `test_no_integration[False-False]` and
`test_binary_content_encoding_image_url[True-False]`. All 10 litellm
failures I looked at on master have the same cause:
```
/usr/lib/python3.10/threading.py:1011: in _bootstrap_inner
_sys.settrace(_trace_hook)
E RuntimeError: Cannot install a trace function while another trace function is being installed
```
**Root cause**
- On CPython < 3.12, `_PyEval_SetTrace` uses a `static int reentrant`
flag shared by the whole process. The flag stays set while audit hooks
run
([python/cpython#98257](python/cpython#98257),
fixed in 3.12).
- `filelock` 3.30.0 through 4.0.2 registers a Python audit hook
(`sys.addaudithook(_audit_fork_safety)`) when it is imported. The py3.10
litellm envs pin `filelock==3.32.4`, and `huggingface_hub` imports it
during the first test.
- pytest-cov calls `threading.settrace`, so every new thread calls
`sys.settrace()` while it starts up. If the GIL switches while another
thread is inside filelock's hook, the new thread's `settrace()` raises
and the thread dies before it runs.
- The tests create a new single-worker executor
(`reset_litellm_executor`), and LiteLLM finishes the `gen_ai.chat` span
in a success callback submitted to it. A dead worker means the callback
never runs, so the test gets `len(chat_spans) == 0`.
**Fix**
`filelock` 4.0.3 fixed this upstream: it no longer registers the hook on
CPython < 3.12 (`_SETTRACE_SURVIVES_AUDIT_HOOKS`). This PR adds a
`py3.10,py3.11` rule, `filelock>=4.0.3`, to the litellm config, then
regenerates `tox.ini` with `scripts/generate-test-files.sh --only
litellm`.
- Only the four `py3.10-litellm-*` blocks change; they now pin
`filelock==4.1.0`. py3.12 and later are unaffected and unchanged.
- Re-resolving those four blocks also moved other transitive pins, for
example `anthropic` 1.0.0 → 1.13.0.
- `releases.jsonl` is unchanged. `--only` would have pruned it, so I
restored it. `package_dependencies.jsonl` only gains the 3 new
resolution entries.
No retries, sleeps, skipped tests or weaker assertions.
**Reproduction:** getsentry/repro#60
It is a standalone script with no litellm and no pytest. On Python 3.10
with filelock 3.32.4, it lost 14 of 32,000 callbacks in one run, and 3
of 5 shorter runs (16,000 callbacks each) reproduced it. Under the same
conditions it lost 0 of 32,000 with filelock 4.0.3, and 0 on Python
3.12.
Original suite, run locally (`py3.10-litellm-v1.83.14`, with coverage):
- With default settings: 0 of 20 runs failed.
- With GIL switching set to happen more often
(`sys.setswitchinterval(1e-6)` via a `-p` plugin; no test code changed):
4 of 20 runs hit the `RuntimeError`, and 3 of 20 failed.
This is a **draft** until validation of the fix against both the repro
and the original suite is posted here.
Possible follow-up, not changed here: `py3.11-anthropic-v0.16.0` also
pins `filelock==3.32.4`.
#### Issues
* resolves: #7014
* Linear: PY-2660
<!-- junior-request-attribution:start -->
via Junior system actor `event`.
<!-- junior-request-attribution:end -->
<!-- junior-session-footer:start -->
<!--
junior-conversation-id:agent-dispatch%3Adispatch_6df7374375b237816e130534b6439833
-->
--
[View Junior
Session](https://junior-prod.sentry.dev/conversations/agent-dispatch%3Adispatch_6df7374375b237816e130534b6439833)
[[Sentry]](https://sentry.sentry.io/explore/conversations/agent-dispatch%3Adispatch_6df7374375b237816e130534b6439833/?project=4510944073809921)
<!-- junior-session-footer:end -->
Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com>1 parent 2ffd383 commit c96bbef
3 files changed
Lines changed: 196 additions & 183 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
287 | 287 | | |
288 | 288 | | |
289 | 289 | | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
290 | 299 | | |
291 | 300 | | |
292 | 301 | | |
| |||
0 commit comments