BrowserFetcher: networkidle navigation hangs and non-navigable (.md) URLs abort scrapes

[evilhamsterman]

Summary

BrowserFetcher has two issues that surface when the browser fallback is exercised on real-world sites (e.g. anything behind a Cloudflare Managed Challenge, or Read-the-Docs sites reached via llms.txt markdown variants):

1. networkidle navigation never settles → page.goto times out. BrowserFetcher.fetch() navigates with waitUntil: "networkidle". Many sites keep background connections open indefinitely (analytics, Cloudflare telemetry, websockets), so the network never goes idle and page.goto rejects with TimeoutError: page.goto: Timeout 30000ms exceeded even though the document loaded almost immediately. (Note: HtmlPlaywrightMiddleware already avoids this by gating on "load" and treating networkidle as best-effort — BrowserFetcher doesn't.)

2. Non-navigable resources (.md, JSON, plain text) crash with ERR_INVALID_ARGUMENT. The llms.txt discovery feature seeds Markdown (.md) URL variants. When such a URL is fetched via the browser fallback (e.g. it returned a Cloudflare 403/challenge over HTTP), page.goto on a Markdown resource makes the browser begin a download and rejects with net::ERR_INVALID_ARGUMENT. Because llms.txt seeds these at depth 0, and BaseScraperStrategy treats any depth-0 failure as fatal, a single such URL aborts the entire scrape job.

Repro

Scrape a Cloudflare-challenged docs site that publishes llms.txt with .md variants (e.g. https://docs.vyos.io/en/1.5/). Observed:

❌ Browser fetch failed for .../automation/index.md: page.goto: net::ERR_INVALID_ARGUMENT
   - navigating to ".../automation/index.md", waiting until "networkidle"
❌ Job failed: ... ScraperError: Browser fetch failed for .../automation/index.md: ...ERR_INVALID_ARGUMENT

The job dies within ~30s on the first non-navigable seed; on sites where it gets further, pages intermittently fail with networkidle timeouts.

Proposed fix

• Gate page.goto on "load" instead of "networkidle", then wait for networkidle only on a best-effort basis (mirrors HtmlPlaywrightMiddleware).
• When page.goto rejects with ERR_INVALID_ARGUMENT / ERR_ABORTED (non-navigable resource), fall back to the browser context's request API (page.request.get), which reuses cookies (so any anti-bot clearance carries over) but does not try to render the response. A still-blocked fetch surfaces as a retryable ScraperError instead of crashing the job.

PR with this fix + tests to follow.