fix(http): bound the on-disk response cache

[afonsojanu]

Fixes #166

Summary

_DiskCache.put() only ever wrote a new file — nothing pruned it — so a key fetched once and never again (e.g. a large crawl, or a long-running MCP server with cache_dir set) stayed on disk forever. The in-memory _ResponseCache already has an LRU cap for exactly this reason (cache_max_size); the disk tier had no equivalent.

Fix

put() now prunes after every write:

1. Sweep any expired entry first, so one that is never re-requested doesn't linger — unlike get()'s lazy expiry, which only reaps a key that's actually asked for again.
2. If the directory still exceeds max_size, delete the oldest files by mtime until it doesn't.

Both _DiskCache.__init__ and put() take the cap, mirroring _ResponseCache's existing put(key, resp, max_size=None) per-call-override pattern. Added ScraperConfig.cache_dir_max_size (default 512, matching cache_max_size) and threaded it through both the sync and async _cache_put(), which also now pass cache_ttl through so the sweep has something to check against.

Testing

Four new tests in TestHttpClientCaching, covering the issue's own three required cases plus the per-call override:

• test_disk_cache_evicts_oldest_past_max_size — put 20, cap 5 → exactly 5 remain, the 5 most recently written.
• test_disk_cache_sweeps_an_expired_entry_even_if_never_reread — an entry that expires between two put() calls to different keys is gone after the second, without ever being get()'d.
• test_disk_cache_prune_failure_does_not_raise — the cache dir is removed out from under the cache; the following put() doesn't raise.
• test_disk_cache_put_max_size_overrides_the_instance_default — a per-call max_size on put() wins over the instance default.

Confirmed all four fail against the pre-fix code (git stash on the three source files, re-ran): each raises TypeError for the parameter that doesn't exist yet.

uv run pytest tests/ -q
# 530 passed, 9 skipped, 19 deselected
uv run ruff check src/pyscrappy/core/http.py src/pyscrappy/core/async_http.py src/pyscrappy/core/config.py tests/test_core/test_http.py
# All checks passed!

mypy reports 6 pre-existing errors in http.py/async_http.py about Response | _StealthResponse typing at _cache_put()'s existing call sites — identical set, same messages, only shifted line numbers, confirmed by running mypy against the unmodified files. Unrelated to this change.

[vedaant00]

Thanks @afonsojanu, clean fix for #166. Right call: get() only reaps re-requested keys, so the disk tier needed prune-on-write. Sweep-then-trim, per-call max_size, best-effort, all match the rest of the cache.

Resolved the conflict for you (kept your tests + the new TestObservabilityHooks) and pushed. 583 passed, ruff clean. Merging.