feat: exclude 404 pages from summary totals and the directory tree

[YusukeHirao]

Summary

A 404 URL has no page behind it, so 404 rows no longer inflate any count. Inventory-driven crawls can seed tens of thousands of URLs that turn out not to exist, and until now those input mistakes dominated every summary number and the directory tree.

query

• getSummary excludes status = 404 rows (regardless of source) from totalPages / internalPages / externalPages / internalContents / externalContents and contentTypeDistribution.
• statusDistribution keeps 404s, now split into two rows: the plain 404 row counts fix-target broken pages (crawled / inventory-discovered), while source = 'inventory-seed' rows (input mistakes from crawl --inventory lists) move to a trailing row marked inventorySeed: true — ordered after every regular status, before the null-status trailer. Ordering lives in a single tier-based comparator.
• metadataFulfillment applies a deliberately asymmetric, narrower rule: only inventory-seed 404s leave the denominator. A fix-target 404 still owes its metadata, so it keeps dragging the rates until fixed.
• buildDirectoryTreeRows drops 404 rows before host qualification: no counts, no viewer_directory_pages membership, no node for a directory whose pages are all 404s, no tree for a host whose internal rows are all 404s. DirectoryTreeSourceRow gains status for this.
• VIEWER_READ_MODEL_SCHEMA_VERSION bumped to 24 so read models built with 404-inclusive counts are treated as stale (live fallback / [] until viewer-build, per the existing mechanism).
• NULL-status legacy rows are never treated as 404s; source is typed as PageSource so a literal typo cannot silently never-match.

viewer

• The summary status histogram renders the seed row as 404 (inventory-seed). Key/label derivation is extracted into the pure, unit-tested buildStatusRowDescriptor (the two 404 rows share status, so keying on status alone would collide in React's reconciliation).

docs

• MCP get_summary tool description documents the exclusion rules for MCP clients interpreting the counts.
• ARCHITECTURE.md indexes the cross-feature "404 no-count" invariant (details stay in JSDoc).

Behavior notes

• Everything flows through getSummary / the read model, so viewer, MCP (open_archive / get_summary), and nitpicker query summary all change together.
• The Pages view, link graph, inbound links, and broken-links audit still include 404s — the Pages status filter remains the place to locate them.
• JSON output shape: StatusCount gains the optional inventorySeed?: true field (v0.x, no compatibility shim).

Tests

• getSummary: dedicated 404-exclusion describe (totals, provenance split with hardcoded literals, asymmetric fulfillment denominator pinned as 1/2 vs 1/1 vs 1/3), plus a null-status ordering describe pinning [numeric…, seed, null].
• Legacy counting behavior is preserved as a regression net: existing fixtures that used 404 to mean "an errored-but-existing page still counts" now use 403, keeping their original expectations intact.
• buildDirectoryTreeRows: unit tests for full exclusion, all-404 directories, host qualification, and NULL-status passthrough; buildViewerReadModel integration test proves the exclusion end-to-end (SELECT → mapping → builder) and that the page stays a normal viewer_pages row.
• getViewerSummary: round-trip test proving inventorySeed survives viewer_summary.status_json persistence.
• buildStatusRowDescriptor: hardcoded key/label expectations including the collision-avoidance suffix.

yarn lint / yarn lint:eslint:check / yarn build / yarn test all green.

🤖 Generated with Claude Code