web_fetch injects raw PDF binary into agent context instead of extracting text

[jlacour-git]

Summary

web_fetch decodes every HTTP response body via resp.text(), which turns a
PDF into a String of raw, FlateDecode-compressed bytes and passes it straight
into the agent's context. A single medium PDF (a restaurant menu in our case)
produced ~617 K characters of binary. This overflows the model's context window
and — with a local model — makes it emit garbled tokens that persist across the
whole thread, since the poisoned history is re-sent on every subsequent turn.

The Content-Type is read (web_fetch.rs, ~L115) but never used except for the
HTML check, so application/pdf falls through to the raw-body branch.

Reproduction

1. Point any agent at a PDF URL and ask it to read the content, e.g.
   web_fetch https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf
2. Observe the tool result contains %PDF-1.x … stream … endstream raw bytes.
3. With a larger PDF, the request exceeds the context window
   (exceed_context_size_error) and following messages return corrupted output.

Second, related issue at the same spot

The size guard at web_fetch.rs L106 only fires when the server sends a
Content-Length header. For chunked responses without it, resp.text()
buffers the entire body into memory unbounded before max_chars truncation —
a memory-exhaustion vector.

Note on the bundled pdf-reader skill

OpenFang ships a pdf-reader skill, but it is prompt-only: it can't help here,
because the raw binary has already entered the context before any skill logic
runs. The fix has to happen at the tool layer, before the body reaches the model.

Proposed fix

Read the body as a size-capped byte stream and route by Content-Type:
application/pdf → text extraction (pure-Rust pdf-extract, no new system deps),
everything else → lossy UTF-8. Binary is never passed through as a string, and
the streaming cap also closes the chunked-response memory issue.

We already have this implemented and running against v0.6.9 (acf2587):
cargo test --workspace, cargo clippy -D warnings, and cargo fmt --check
all pass, plus 4 new tests. Happy to open a PR if you'd welcome it.