diff --git a/.github/workflows/viewer-e2e.yml b/.github/workflows/viewer-e2e.yml index 252db540..9301a3ad 100644 --- a/.github/workflows/viewer-e2e.yml +++ b/.github/workflows/viewer-e2e.yml @@ -44,3 +44,4 @@ jobs: - run: yarn workspace @nitpicker/viewer test:e2e:stub - run: yarn workspace @nitpicker/viewer test:e2e:directory-tree - run: yarn workspace @nitpicker/viewer test:e2e:template-clusters + - run: yarn workspace @nitpicker/viewer test:e2e:inbound-links diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index ac9441e3..6771d365 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -66,7 +66,7 @@ - **network_outages**: オペレータ側ネットワーク断（と疑われた区間）の append-only 履歴。`started_at`（検出ウィンドウ内の最古エラーまで遡及）/ `detected_at` / `ended_at`（復旧まで NULL）/ `probe_host` / `trigger_error_count` / `trigger_host_count`。索引なし（1 クロールあたり数件、消費側は全行をメモリに読んで `isWithinOutageWindow` で判定する）。`ended_at` は復旧確定 or 次回 writer open 時の打ち切りで一度だけ UPDATE される（`crawler/src/is-within-outage-window.ts`、`archive/db-ops/outages/`） - **console_log_items / page_console_logs**（issue #228、beholder `ScrapeResult.consoleLogs`）: 2 テーブル構成。`console_log_items` は `text_refs` / `json_refs` / `url_refs` と同じ内容ハッシュ dedup の辞書（`hash` が `type` + `text` + `args` JSON + `location` + `stack` の canonical タプルの SHA-256）で、全ページ横断で 1 内容 1 行 — 同一フレームワーク警告が全ページで出るケースでもストレージは線形に増えない。`page_console_logs` は `(page_id, console_log_id, ts)` の 1 出現 1 行のエッジテーブルで、同じページが同じメッセージを 3 回出せば 3 行になる。書き込みは `replaceConsoleLogs`（`crawler/src/archive/db-ops/console-logs/`）が 1 トランザクションで `DELETE FROM page_console_logs WHERE page_id = ?` → 辞書 upsert → INSERT を行う Scoped-Replace（`anchor_edges` と同じ分類、下記 4 分類参照）。空 `entries` では呼び出し自体をスキップする非空ガードは crawler 側（`Crawler#handleConsoleLogs`）が担う。`page_meta.console_error_count`（`pageerror`+`error` の occurrence 数）は `tag_count` と同じ denormalised aggregate だが、書き込みのアトミック性は異なる — `tag_count` は `insertPage` が `page_meta` 行自体を作る INSERT に同梱するのに対し、`console_error_count` は `replaceConsoleLogs` が entries から直接計算して**別トランザクション**で UPDATE する（crawler の `'page'` イベントと `'consoleLogs'` イベントは別々の WriteQueue タスクなため）。両者の間でプロセスが落ちると、その 1 ページの `console_error_count` は列デフォルト値のまま永続的に取り残される（自己修復しない。page_meta 行が無い `'skipped'`/`'error'` スクレイプでは UPDATE が 0 行に落ちるだけで無害）。読み取りは read model 非経由のライブ集計（`query/src/list-console-logs.ts` / `get-page-console-logs.ts` / `count-console-logs-by-type.ts`）— 新機能で前身データが無いため `viewer_header_checks` 級の read-model + fast-path 二層は採用せず `getViolations` 相当のシンプル構成。`viewer_pages.console_error_count` / `viewer_summary.console_json` のみ read model にコピーする（ソート・ダッシュボード用） - **リダイレクト**: 独立テーブルなし。`content_items.redirect_dest_id` を書き込み時（`linkRedirectSources`）に常に最終宛先まで pre-flatten し、読み取りは `COALESCE(target.redirect_dest_id, target.id)` の 1 ホップ（read 時に chain-walk しない） -- **content_items.alias_of_id**: URL 正規化で同一ページとみなせる行同士を統合する自己参照 FK（`redirect_dest_id` と同形、`DEFERRABLE INITIALLY DEFERRED`）。実際の 3xx を要件としない点で redirect と異なる。判定は 2 階層とも `page_meta.title_text_id` 一致が AND 条件: Tier A（`computeTierAAliasKey` — scheme 等価・host 大文字小文字・`/index.{ext}` 表記ゆれ、URL 文字列のみで決定的）と Tier B（`computeTierBAliasKey` — さらに末尾スラッシュ差異のみ、`page_meta.body_hash` 一致も必須）。両ティアは決定的キー関数の完全一致でグルーピングするため各ティア単体は推移的だが、2 種の関係の合成は自動では推移的にならず、Union-Find（`viewer-read-model/backfill-alias-of-id.ts`）で連結成分の閉包を取る（`compareUrlSortKeys` のペア比較が推移律を保証しないのとは別の理由・別の設計）。代表選定は「グループ内の他メンバーを指す canonical 数最多」→「最短 URL」→「文字列昇順」。列自体は `body_hash` と同じ自己修復マイグレーション（`migrate-content-items-alias-of-id.ts`、索引は DDL でなくマイグレーション側で作成 — 理由は body_hash の索引バグと同じ）で追加されるが、値の計算は backfill ではなく `viewer-build` 実行毎のフルリコンピュート（`backfillAliasOfId`、`body_hash` backfill の直後 — Tier B が計算済み body_hash に依存するため）。read-only 接続では列追加が走らないため、read-only 経路からも呼ばれる全クエリ（`get-summary.ts` / `list-pages.ts` / `get-link-graph.ts` / `get-page-detail.ts` / `find-duplicates.ts` / `find-mismatches.ts` / `check-headers.ts` / `list-links.ts` / `compute-isolated-clusters.ts`）が冒頭で `requireAliasOfIdColumn` を呼び、列が無ければ `viewer-build` 実行を促すエラーを投げる。`build-viewer-read-model.ts` / `compute-anchor-fact-rows.ts` はこのガードを呼ばない — 書き込み可能接続（`ensureViewerReadModel` 経由）でのみ実行され、その時点で `migrateContentItemsAliasOfId` 適用済みが保証されるため不要。適用パターンは箇所ごとに `redirect_dest_id` の既存方針を踏襲: 除外専用の `get-link-graph.ts` / `compute-isolated-clusters.ts` の候補集合は drop、`list-links.ts` / `compute-anchor-fact-rows.ts` の宛先解決は `COALESCE` で resolve。`list-pages.ts` の `urlPattern` フィルターと `get-page-detail.ts` の URL 検索は、redirect 元 URL・alias メンバー URL のどちらで検索してもヒットするよう、`redirect_dest_id` と `alias_of_id` を対等に解決する（`list-pages.ts` は各カラムを個別 `IN` サブクエリにしてから `UNION ALL` で束ねる — 1 つの `OR` に畳むとフルスキャンに落ちる負の知識は前述のとおり）。両者は別の関係（redirect は実観測の 3xx、alias は本文/URL 形状からの推論）だが「このURLの実体は別の行」という意味では同じ扱いにする。 +- **content_items.alias_of_id**: URL 正規化で同一ページとみなせる行同士を統合する自己参照 FK（`redirect_dest_id` と同形、`DEFERRABLE INITIALLY DEFERRED`）。実際の 3xx を要件としない点で redirect と異なる。判定は 2 階層とも `page_meta.title_text_id` 一致が AND 条件: Tier A（`computeTierAAliasKey` — scheme 等価・host 大文字小文字・`/index.{ext}` 表記ゆれ、URL 文字列のみで決定的）と Tier B（`computeTierBAliasKey` — さらに末尾スラッシュ差異のみ、`page_meta.body_hash` 一致も必須）。両ティアは決定的キー関数の完全一致でグルーピングするため各ティア単体は推移的だが、2 種の関係の合成は自動では推移的にならず、Union-Find（`viewer-read-model/backfill-alias-of-id.ts`）で連結成分の閉包を取る（`compareUrlSortKeys` のペア比較が推移律を保証しないのとは別の理由・別の設計）。代表選定は「グループ内の他メンバーを指す canonical 数最多」→「最短 URL」→「文字列昇順」。列自体は `body_hash` と同じ自己修復マイグレーション（`migrate-content-items-alias-of-id.ts`、索引は DDL でなくマイグレーション側で作成 — 理由は body_hash の索引バグと同じ）で追加されるが、値の計算は backfill ではなく `viewer-build` 実行毎のフルリコンピュート（`backfillAliasOfId`、`body_hash` backfill の直後 — Tier B が計算済み body_hash に依存するため）。read-only 接続では列追加が走らないため、read-only 経路からも呼ばれる全クエリ（`get-summary.ts` / `list-pages.ts` / `get-link-graph.ts` / `get-page-detail.ts` / `list-inbound-links.ts` / `find-duplicates.ts` / `find-mismatches.ts` / `check-headers.ts` / `list-links.ts` / `compute-isolated-clusters.ts`）が冒頭で `requireAliasOfIdColumn` を呼び、列が無ければ `viewer-build` 実行を促すエラーを投げる。`build-viewer-read-model.ts` / `compute-anchor-fact-rows.ts` はこのガードを呼ばない — 書き込み可能接続（`ensureViewerReadModel` 経由）でのみ実行され、その時点で `migrateContentItemsAliasOfId` 適用済みが保証されるため不要。適用パターンは箇所ごとに `redirect_dest_id` の既存方針を踏襲: 除外専用の `get-link-graph.ts` / `compute-isolated-clusters.ts` の候補集合は drop、`list-links.ts` / `compute-anchor-fact-rows.ts` の宛先解決は `COALESCE` で resolve。`list-pages.ts` の `urlPattern` フィルターと `get-page-detail.ts` の URL 検索は、redirect 元 URL・alias メンバー URL のどちらで検索してもヒットするよう、`redirect_dest_id` と `alias_of_id` を対等に解決する（`list-pages.ts` は各カラムを個別 `IN` サブクエリにしてから `UNION ALL` で束ねる — 1 つの `OR` に畳むとフルスキャンに落ちる負の知識は前述のとおり）。両者は別の関係（redirect は実観測の 3xx、alias は本文/URL 形状からの推論）だが「このURLの実体は別の行」という意味では同じ扱いにする。 - **多段解決が必要な理由**: `backfillAliasOfId` の候補選定は redirect _元_（`redirect_dest_id IS NOT NULL`）だけを候補から除外し、redirect _先_ は除外しない。そのため「A が B に redirect し、B 自身が C の alias メンバー」という 2 段の連鎖が起こりうる（`redirect_dest_id` 自体は書き込み時に最終宛先まで pre-flatten 済みなので、追加で辿る必要があるのは `alias_of_id` の 1 ホップのみ）。単発行の `get-page-detail.ts` は `resolveAliasAndRedirectChain`（`query/src/resolve-alias-and-redirect-chain.ts`）で `redirect_dest_id`/`alias_of_id` を交互に 1 ホップずつ辿る（サイクル検出付き、DB 往復は許容 — 1 ページ分のみ）。`anchor_edges` を一括処理する `list-links.ts` / `compute-anchor-fact-rows.ts` は DB 往復を増やせないため、`canonical`（redirect 先）自身の `alias_of_id` を解決するもう 1 段の JOIN（`canonical_alias`）を追加し、`COALESCE(canonical_alias.*, canonical.*, alias_canonical.*, dest.*)` で優先順位を表現する（`compute-isolated-clusters.ts` は元々 `redirectMap` を Union-Find 前に `resolveRedirectChain` で辿り切る設計だったため、この多段化の影響を受けない） - **reader / writer の対称性**: reader も writer も同じ 0.13 entity / ref テーブルを使う（issue #196 で writer 切替済み、ズレなし）。読み取りは flat な `DB_Page` / `DB_Resource` 形状を join で再構築する（`db-ops/pages/read/build-page-query.ts` + `reconstruct-page-rows.ts`、`db-ops/resources/` の対応ファイル）。fresh / migrated アーカイブの FK 宣言は `content_items(id)` に統一済み（migrated 側は `migrate-to-0.13.mjs` の rename-copy-drop が保証、`PRAGMA foreign_key_check` 0 件を `verify-migration/check-foreign-key-integrity.ts` が最終検証する） - **viewer read model（`viewer_*` テーブル群）**: `buildViewerReadModel`（`query/src/viewer-read-model/build-viewer-read-model.ts`）が構築する読み取り専用の事前計算層。pages / summary / error-kinds / resources / images / header-checks / duplicates / mismatches / anchor-facts / directory-tree をカバーし、各機能は `get-*-fast-path.ts` で read model（fast path）と legacy SQL の二層 dispatch を行う @@ -82,16 +82,18 @@ - **`redirectPaths` の `slice(1)` を外さない** — follow-redirects の `res.redirects[0]` はクエリ落ちした元 URL であり、外すとクエリ違いの別ページが同一視され消失する（`crawler/src/crawler/fetch-destination.ts`。E2E: `/query-distinct/`） - **「ページか」は content-type で判定する。`isTarget` で判定しない** — `isTarget` は「in-scope なクロール対象か」であり、in-scope な PDF も `isTarget=1`（`crawler/src/archive/normalize-content-type.ts`、`query/src/list-pages.ts`） - **被リンクは redirect 透過解決するが、発リンクは解決しない** — 発リンク側の raw な指し先は「古い URL にリンクしている」という監査シグナル。この非対称性を「統一」しないこと（`database.ts`） -- **被リンク系の集約粒度は referrer 単位で揃える** — `getPageDetail.inboundLinks`（referrer で GROUP BY）と `listExternalLinks.referrerCount`（`COUNT(DISTINCT source.id)`）は同一粒度。片方だけ anchor 単位に変えると外部リンク一覧の参照元数と Page Detail の被リンク件数が食い違う（`query/src/list-external-links.ts`、`get-page-detail.ts`） +- **被リンク系の集約粒度は referrer 単位で揃える** — `listInboundLinks`（`viewer_anchor_facts` は `(source_page_id, dest_page_id)` で一意）と `listExternalLinks.referrerCount`（`COUNT(DISTINCT source.id)`）は同一粒度。片方だけ anchor 単位に変えると外部リンク一覧の参照元数と被リンク件数が食い違う（`query/src/list-external-links.ts`、`list-inbound-links.ts`）。`getPageDetail` はこの集約自体を持たない（inbound links 全体を `listInboundLinks` に切り出し済み、issue #235） - **URL natural-sort comparator は推移律を保証しない** — 重複排除は `compareUrlSortKeys(...) === 0` ではなく `original` 文字列の完全一致で行い、`viewer_url_sort_keys` への INSERT は `onConflict('url').ignore()` を fail-safe に使う。viewer 起動時ソートは外部マージソート（メモリよりチャンクサイズ優先）で、結果は tar-cache 配下に JSONL ストリーミング永続化（`query/src/external-url-sort.ts`、`merge-sorted-url-chunks.ts`、`url-sort-temp-table.ts`、`viewer/src/url-sort-cache.ts`） - **`listPages` 系は `scraped = 1` 前提（受容済みギャップ）** — 除外されて一度も取得されていない URL を一括列挙する手段は意図的に無い（旧 `listPageLinks` の廃止に伴う）。URL 既知なら `getPageDetail`（`isSkipped` / `skipReason` を返す）、一括把握は `query error-kinds` か `pages` テーブル直クエリで行う - **`write()` に到達する writer 経路は必ず `Archive.open({ openPluginData: true })` で開く** — `open()` の既定は `db.sqlite` だけを tmpDir に展開し、`write()` は tmpDir を丸ごと再 tar 化する。`openPluginData` を渡し忘れて再クロール（`append` / `inventory` / `retryFailed`）すると、`db.sqlite` 以外の既存 tar エントリ（analyze の `setData` 出力、保存済み inventory ソースリスト）は展開されないまま書き戻され、黙って消える（`crawler-orchestrator.ts` の各 static メソッド、`cli/src/commands/viewer-build.ts`） - **read-only open は viewer read model を一切 build しない** — on-open opportunistic build は read クエリの長時間ブロックとタールキャッシュ破損の実害があり削除済み（#177）。build 経路は **crawl 完了時（`CrawlerOrchestrator.write()` 直前、`cli/src/crawl/ensure-viewer-read-model-quietly.ts`）と `viewer-build` コマンドの 2 経路のみ**。read model が無い / 古い archive は明示 `viewer-build` まで恒久的に legacy 経路（`query/src/viewer-read-model/is-viewer-read-model-current.ts`） - **stub mode（live crawl）は常に fast path 禁止** — `isViewerReadModelCurrent` はスキーマバージョンしか見ないため、writer 追記中の tmpDir では再開前の古いスナップショットを返しうる。`getSummaryFastPath` 等を経由せず legacy を直呼びする（`viewer/src/summary-cache.ts`、`error-kinds-cache.ts`） -- **legacy fallback が無い機能の guard は `hasViewerReadModel` でなく `isViewerReadModelCurrent`** — 旧スキーマの read model を掴むと `no such table` で 500 になる（directory-tree 3 関数。`query/src/get-directory-tree.ts`） +- **legacy fallback が無い機能の guard は `hasViewerReadModel` でなく `isViewerReadModelCurrent`** — 旧スキーマの read model を掴むと `no such table` で 500 になる（directory-tree 3 関数。`query/src/get-directory-tree.ts`）。`listInboundLinks`（`query/src/list-inbound-links.ts`）も同じ理由で legacy fallback を持たないが、失敗時の扱いは反対: directory-tree は 0 件が無害な `[]` を返すのに対し、被リンク件数は 0 が意味を持つ値なので `requireViewerReadModel` で `throw`（stub mode では viewer route 側が `context.mode` を見てこの関数自体を呼ばない — `viewer/src/routes/register-inbound-links-route.ts`） +- **stub mode の被リンクは fast path 禁止の適用外ではなく「一切提供しない」** — `viewer_anchor_facts` は read-only accessor でも stub ディレクトリでも構築され得ないため（`buildViewerReadModel` は read-only accessor を拒否、`viewer-build` は stub ディレクトリを拒否）、`getPageDetail` は被リンクに一切関与しない設計にして stub mode でも動き続ける（`inboundLinks` フィールド自体が存在しない）。被リンク件数・一覧は viewer では別 API（`/api/pages/inbound-links`）の責務とし、stub mode では `{ available: false }` を返す - **read model の二次索引はデータ全投入後に構築する（index-after-load）** — 投入前に張ると B-tree 維持コストがテーブル成長とともに悪化し、大規模 archive でビルドが完走しない。PK / UNIQUE / `WITHOUT ROWID` は `.onConflict()` が依存するためテーブル定義に残す（`query/src/viewer-read-model/create-viewer-read-model-indexes.ts` / `create-viewer-read-model-tables.ts`） - **evidence-before-indexing** — read model の index は憶測で積まず `EXPLAIN QUERY PLAN` と実測で確定する。既知の落とし穴: 範囲述語（`missing_count > 0`）を先頭列にした index は `ORDER BY url_sort_key` を満たせない（bool 列を先頭に）。default view の `content_category IN ('html','unknown')` が生む TEMP B-TREE は index 追加で解消できない（`list-viewer-header-checks.ts`、`apply-viewer-pages-filters.ts`） - **anchors（0.13: `anchor_edges`）の全走査は `compute-anchor-fact-rows.ts` のみに許す** — build は id 範囲パーティションの `AsyncGenerator` で chunk 化し、chunk 跨ぎの `referrer_count` 合算は JS の Map で持ち回らず SQLite の `ON CONFLICT ... DO UPDATE` に委譲する（どちらも大規模 archive の OOM 回避。`derive-external-link-summary-rows.ts`、`upsert-external-link-rows.ts`） +- **`viewer_anchor_facts.first_text_id` はアンカーテキストの複製ではなく `text_refs(id)` 参照** — `text_refs` は既に同一データベース内にあり、辞書テーブルの内容を複製する読み側メリットが無いため（issue #235、`listInboundLinks` がこの列 1 本で referrer のアンカーテキストを解決し、`anchor_edges` への 2 回目の往復を避ける） - **`page_url_rank` は URL テキストを複製しない** — read model ファミリーで唯一の例外（`viewer_images` は数百万行規模になりうるため）。順位比較は SQLite BINARY 照合に合わせた `compareUrlBinary` を使う（素朴な JS 文字列比較は補助面文字で食い違う。`query/src/viewer-read-model/build-page-url-rank-map.ts`） - **fast path の強制 legacy 条件は関数ごとに異なる** — pages/resources は `urlPattern`/`directory`、broken links は `urlPattern`/`includeRedirectSources`、headers/mismatches は明示 `sortBy`、duplicates のみ強制 legacy なし。各 `get-*-fast-path.ts` の JSDoc が正。duplicates の legacy 経路は実 `group_id` と衝突しない負の sentinel `-(index+1)` を採番し、route は `groupId <= 0` を 404 にする - **既存アーカイブへの列追加は通常 version-gated migration script の役割だが、書き込みパスが単一で nullable な追加列に限り自己修復してよい** diff --git a/packages/@nitpicker/cli/docs/query.md b/packages/@nitpicker/cli/docs/query.md index 6ac05562..8de167bf 100644 --- a/packages/@nitpicker/cli/docs/query.md +++ b/packages/@nitpicker/cli/docs/query.md @@ -28,36 +28,37 @@ npx @nitpicker/cli query ./site.nitpicker page-detail --url https://example.com/ ## サブコマンド一覧 -| サブコマンド | 用途 | -| ---------------------------- | ----------------------------------------------------------------- | -| `summary` | アーカイブ全体の概要統計 | -| `pages` | ページ一覧 | -| `page-detail` | 指定URLのページ詳細 | -| `html` | 指定URLのHTMLスナップショット | -| `links` | broken/externalリンク一覧 | -| `resources` | ネットワークリソース一覧 | -| `images` | 画像一覧と画像品質フィルタ | -| `violations` | 分析プラグインの違反結果 | -| `duplicates` | title/descriptionの重複 | -| `mismatches` | canonical/OGPメタデータの不一致 | -| `headers` | セキュリティヘッダー確認 | -| `resource-referrers` | 指定リソースの参照元ページ | -| `error-kinds` | クロール失敗原因の集計 | -| `pages-by-tag` | Wappalyzerタグに一致するページ | -| `count-pages-by-tag` | Wappalyzerタグに一致するページ数 | -| `pages-by-jsonld-type` | JSON-LD typeに一致するページ | -| `count-pages-by-jsonld-type` | JSON-LD typeに一致するページ数 | -| `tag-inventory` | 検出タグの一覧 | -| `page-jsonld` | 指定URLのJSON-LD | -| `page-jsonld-overview` | 指定URLのJSON-LD概要 | -| `page-tags` | 指定URLの検出タグ | -| `isolated-pages` | inventory由来の完全孤立ページ | -| `isolated-clusters` | inventory由来の孤立クラスタ | -| `get-isolated-cluster` | 指定代表URLの孤立クラスタ詳細 | -| `unused-resources` | 参照元がないinventory由来リソース | -| `inventory-runs` | inventory実行履歴 | -| `console-logs` | 捕捉したconsoleログ・ページエラー（内容ごとに全ページ横断で集約） | -| `page-console-logs` | 指定URLのconsoleログ・ページエラー明細 | +| サブコマンド | 用途 | +| ---------------------------- | ---------------------------------------------------------------------- | +| `summary` | アーカイブ全体の概要統計 | +| `pages` | ページ一覧 | +| `page-detail` | 指定URLのページ詳細 | +| `inbound-links` | 指定URLへの被リンク一覧（referrer単位、cursor/offsetページネーション） | +| `html` | 指定URLのHTMLスナップショット | +| `links` | broken/externalリンク一覧 | +| `resources` | ネットワークリソース一覧 | +| `images` | 画像一覧と画像品質フィルタ | +| `violations` | 分析プラグインの違反結果 | +| `duplicates` | title/descriptionの重複 | +| `mismatches` | canonical/OGPメタデータの不一致 | +| `headers` | セキュリティヘッダー確認 | +| `resource-referrers` | 指定リソースの参照元ページ | +| `error-kinds` | クロール失敗原因の集計 | +| `pages-by-tag` | Wappalyzerタグに一致するページ | +| `count-pages-by-tag` | Wappalyzerタグに一致するページ数 | +| `pages-by-jsonld-type` | JSON-LD typeに一致するページ | +| `count-pages-by-jsonld-type` | JSON-LD typeに一致するページ数 | +| `tag-inventory` | 検出タグの一覧 | +| `page-jsonld` | 指定URLのJSON-LD | +| `page-jsonld-overview` | 指定URLのJSON-LD概要 | +| `page-tags` | 指定URLの検出タグ | +| `isolated-pages` | inventory由来の完全孤立ページ | +| `isolated-clusters` | inventory由来の孤立クラスタ | +| `get-isolated-cluster` | 指定代表URLの孤立クラスタ詳細 | +| `unused-resources` | 参照元がないinventory由来リソース | +| `inventory-runs` | inventory実行履歴 | +| `console-logs` | 捕捉したconsoleログ・ページエラー（内容ごとに全ページ横断で集約） | +| `page-console-logs` | 指定URLのconsoleログ・ページエラー明細 | ## サブコマンド別オプション @@ -104,6 +105,23 @@ npx @nitpicker/cli query ./site.nitpicker page-detail --url https://example.com/ | ---------- | ---------------- | ------------- | | `--url` | string, required | 対象ページURL | +### `inbound-links` + +```sh +npx @nitpicker/cli query ./site.nitpicker inbound-links --url https://example.com/ --pretty +npx @nitpicker/cli query ./site.nitpicker inbound-links --url https://example.com/ --limit 100 --cursor <前回のnextCursor> +``` + +| オプション | 型 | 説明 | +| ---------------- | ---------------- | ------------------------------------------- | +| `--url` | string, required | 対象ページURL | +| `--limit`, `-l` | number | 最大取得件数。`0` で件数（`total`）のみ取得 | +| `--offset`, `-o` | number | スキップ件数（ページ番号ジャンプ用） | +| `--cursor` | string | 前回の結果の `nextCursor`/`prevCursor` | +| `--direction` | string | `--cursor` と併用。`next`（既定）/ `prev` | + +`viewer_anchor_facts` read model のみを読みます。read model が未構築・古い場合は `viewer-build` の実行を促すエラーで失敗します（フォールバックなし）。 + ### `html` ```sh @@ -385,37 +403,39 @@ npx @nitpicker/cli query ./site.nitpicker page-console-logs --url https://exampl ## 全オプション一覧 -| オプション | 型 | 主な用途 | -| ---------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------- | -| `--limit`, `-l` | number | ページネーション | -| `--offset`, `-o` | number | ページネーション | -| `--url` | string | `page-detail` / `html` / `resource-referrers` / `page-jsonld` / `page-jsonld-overview` / `page-tags` / `page-console-logs` | -| `--status` | number | `pages` | -| `--status-min` | number | `pages` | -| `--status-max` | number | `pages` | -| `--is-external` | boolean | `pages` / `resources` | -| `--missing-title` | boolean | `pages` | -| `--missing-description` | boolean | `pages` | -| `--noindex` | boolean | `pages` | -| `--url-pattern` | string | `pages` / `images` | -| `--directory` | string | `pages` | -| `--sort-by` | string | `pages` / `console-logs` | -| `--sort-order` | string | `pages` / `console-logs` | -| `--type` | string | `links` / `mismatches` / JSON-LD type系 / `console-logs` | -| `--content-type` | string | `resources` | -| `--content-type-category` | string | `pages` | -| `--missing-alt` | boolean | `images` | -| `--missing-dimensions` | boolean | `images` | -| `--oversized-threshold` | number | `images` | -| `--validator` | string | `violations` | -| `--severity` | string | `violations` | -| `--rule` | string | `violations` | -| `--field` | string | `duplicates` | -| `--missing-only` | boolean | `headers` | -| `--max-length` | number | `html` | -| `--provider` | string | tag系 | -| `--external-id` | string | tag系 | -| `--full` | boolean | `page-jsonld` | -| `--representative-url` | string | `get-isolated-cluster` | -| `--include-redirect-sources` | boolean | `links` | -| `--pretty` | boolean | JSON整形 | +| オプション | 型 | 主な用途 | +| ---------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------- | +| `--limit`, `-l` | number | ページネーション | +| `--offset`, `-o` | number | ページネーション | +| `--cursor` | string | `resource-referrers` / `duplicates` / `mismatches` / `inbound-links` | +| `--direction` | string | `duplicates` / `mismatches` / `inbound-links`（`--cursor` と併用） | +| `--url` | string | `page-detail` / `inbound-links` / `html` / `resource-referrers` / `page-jsonld` / `page-jsonld-overview` / `page-tags` / `page-console-logs` | +| `--status` | number | `pages` | +| `--status-min` | number | `pages` | +| `--status-max` | number | `pages` | +| `--is-external` | boolean | `pages` / `resources` | +| `--missing-title` | boolean | `pages` | +| `--missing-description` | boolean | `pages` | +| `--noindex` | boolean | `pages` | +| `--url-pattern` | string | `pages` / `images` | +| `--directory` | string | `pages` | +| `--sort-by` | string | `pages` / `console-logs` | +| `--sort-order` | string | `pages` / `console-logs` | +| `--type` | string | `links` / `mismatches` / JSON-LD type系 / `console-logs` | +| `--content-type` | string | `resources` | +| `--content-type-category` | string | `pages` | +| `--missing-alt` | boolean | `images` | +| `--missing-dimensions` | boolean | `images` | +| `--oversized-threshold` | number | `images` | +| `--validator` | string | `violations` | +| `--severity` | string | `violations` | +| `--rule` | string | `violations` | +| `--field` | string | `duplicates` | +| `--missing-only` | boolean | `headers` | +| `--max-length` | number | `html` | +| `--provider` | string | tag系 | +| `--external-id` | string | tag系 | +| `--full` | boolean | `page-jsonld` | +| `--representative-url` | string | `get-isolated-cluster` | +| `--include-redirect-sources` | boolean | `links` | +| `--pretty` | boolean | JSON整形 | diff --git a/packages/@nitpicker/cli/src/commands/query.ts b/packages/@nitpicker/cli/src/commands/query.ts index 7b58f135..531ba78a 100644 --- a/packages/@nitpicker/cli/src/commands/query.ts +++ b/packages/@nitpicker/cli/src/commands/query.ts @@ -28,11 +28,11 @@ export const commandDef = { }, cursor: { type: 'string', - desc: 'Opaque pagination cursor from a previous result (resource-referrers, duplicates, mismatches)', + desc: 'Opaque pagination cursor from a previous result (resource-referrers, inbound-links, duplicates, mismatches)', }, direction: { type: 'string', - desc: 'Direction to walk from --cursor: next (default) or prev (duplicates, mismatches)', + desc: 'Direction to walk from --cursor: next (default) or prev (inbound-links, duplicates, mismatches)', }, pagesLimit: { type: 'number', @@ -40,7 +40,7 @@ export const commandDef = { }, url: { type: 'string', - desc: 'Target URL for page-detail, html, resource-referrers, or page-console-logs queries', + desc: 'Target URL for page-detail, inbound-links, html, resource-referrers, or page-console-logs queries', }, status: { type: 'number', diff --git a/packages/@nitpicker/cli/src/query/dispatch-query.spec.ts b/packages/@nitpicker/cli/src/query/dispatch-query.spec.ts index a1cdbdbf..b45c224b 100644 --- a/packages/@nitpicker/cli/src/query/dispatch-query.spec.ts +++ b/packages/@nitpicker/cli/src/query/dispatch-query.spec.ts @@ -11,6 +11,15 @@ vi.mock('@nitpicker/query', () => ({ .mockResolvedValue({ total: 0, channelSource: 'none', groups: [] }), listPages: vi.fn().mockResolvedValue({ items: [], total: 0, offset: 0, limit: 100 }), getPageDetail: vi.fn().mockResolvedValue({ url: 'https://example.com', status: 200 }), + listInboundLinks: vi.fn().mockResolvedValue({ + url: 'https://example.com', + items: [], + total: 0, + limit: 100, + offset: 0, + nextCursor: null, + prevCursor: null, + }), getPageHtml: vi.fn().mockResolvedValue({ html: '', truncated: false }), listLinks: vi.fn().mockResolvedValue({ items: [], total: 0 }), listResources: vi @@ -112,6 +121,58 @@ describe('dispatchQuery', () => { ).rejects.toThrow('Page not found: https://missing.example.com'); }); + it('dispatches inbound-links sub-command', async () => { + const { listInboundLinks } = await import('@nitpicker/query'); + const result = await dispatchQuery(mockAccessor, 'inbound-links', { + url: 'https://example.com', + } as never); + expect(result).toEqual({ + url: 'https://example.com', + items: [], + total: 0, + limit: 100, + offset: 0, + nextCursor: null, + prevCursor: null, + }); + expect(listInboundLinks).toHaveBeenCalledWith(mockAccessor, { + url: 'https://example.com', + limit: undefined, + offset: undefined, + cursor: undefined, + direction: undefined, + }); + }); + + it('dispatches inbound-links sub-command with limit, offset, cursor, direction', async () => { + const { listInboundLinks } = await import('@nitpicker/query'); + await dispatchQuery(mockAccessor, 'inbound-links', { + url: 'https://example.com', + limit: 10, + offset: 20, + cursor: 'abc', + direction: 'prev', + } as never); + expect(listInboundLinks).toHaveBeenCalledWith(mockAccessor, { + url: 'https://example.com', + limit: 10, + offset: 20, + cursor: 'abc', + direction: 'prev', + }); + }); + + it('throws when inbound-links returns null', async () => { + const { listInboundLinks } = await import('@nitpicker/query'); + vi.mocked(listInboundLinks).mockResolvedValueOnce(null); + + await expect( + dispatchQuery(mockAccessor, 'inbound-links', { + url: 'https://missing.example.com', + } as never), + ).rejects.toThrow('Page not found: https://missing.example.com'); + }); + it('dispatches html sub-command', async () => { const { getPageHtml } = await import('@nitpicker/query'); const result = await dispatchQuery(mockAccessor, 'html', { diff --git a/packages/@nitpicker/cli/src/query/dispatch-query.ts b/packages/@nitpicker/cli/src/query/dispatch-query.ts index a283a3fd..a00af689 100644 --- a/packages/@nitpicker/cli/src/query/dispatch-query.ts +++ b/packages/@nitpicker/cli/src/query/dispatch-query.ts @@ -34,6 +34,7 @@ import { getTagInventory, getViolations, listConsoleLogs, + listInboundLinks, listInventoryRuns, listIsolatedClustersFastPath, listIsolatedPagesFastPath, @@ -85,6 +86,26 @@ export async function dispatchQuery( } return result; } + case 'inbound-links': { + const { url, limit, offset, cursor, direction } = options as { + url: string; + limit?: number; + offset?: number; + cursor?: string; + direction?: 'next' | 'prev'; + }; + const result = await listInboundLinks(accessor, { + url, + limit, + offset, + cursor, + direction, + }); + if (!result) { + throw new Error(`Page not found: ${url}`); + } + return result; + } case 'html': { const { url, maxLength } = options as { url: string; maxLength?: number }; const result = await getPageHtml(accessor, url, maxLength); diff --git a/packages/@nitpicker/cli/src/query/map-flags-to-query-options.spec.ts b/packages/@nitpicker/cli/src/query/map-flags-to-query-options.spec.ts index ca26c866..983cfd7e 100644 --- a/packages/@nitpicker/cli/src/query/map-flags-to-query-options.spec.ts +++ b/packages/@nitpicker/cli/src/query/map-flags-to-query-options.spec.ts @@ -79,6 +79,39 @@ describe('mapFlagsToQueryOptions', () => { ); }); + it('requires --url for inbound-links', () => { + expect(() => mapFlagsToQueryOptions('inbound-links', {})).toThrow( + '--url is required for the inbound-links sub-command', + ); + }); + + it('throws for invalid inbound-links direction', () => { + expect(() => + mapFlagsToQueryOptions('inbound-links', { + url: 'https://example.com', + direction: 'sideways', + }), + ).toThrow('Invalid --direction value'); + }); + + it('returns url/limit/offset/cursor/direction for inbound-links', () => { + expect( + mapFlagsToQueryOptions('inbound-links', { + url: 'https://example.com', + limit: 10, + offset: 20, + cursor: 'abc', + direction: 'prev', + }), + ).toEqual({ + url: 'https://example.com', + limit: 10, + offset: 20, + cursor: 'abc', + direction: 'prev', + }); + }); + it('requires --url for html', () => { expect(() => mapFlagsToQueryOptions('html', {})).toThrow( '--url is required for the html sub-command', diff --git a/packages/@nitpicker/cli/src/query/map-flags-to-query-options.ts b/packages/@nitpicker/cli/src/query/map-flags-to-query-options.ts index 6792333b..5777262f 100644 --- a/packages/@nitpicker/cli/src/query/map-flags-to-query-options.ts +++ b/packages/@nitpicker/cli/src/query/map-flags-to-query-options.ts @@ -83,6 +83,23 @@ export function mapFlagsToQueryOptions( } return { url: flags.url }; } + case 'inbound-links': { + if (!flags.url) { + throw new Error('--url is required for the inbound-links sub-command.'); + } + if (flags.direction != null && !['next', 'prev'].includes(flags.direction)) { + throw new Error( + `Invalid --direction value: ${flags.direction}. Must be one of: next, prev`, + ); + } + return { + url: flags.url, + limit: flags.limit, + offset: flags.offset, + cursor: flags.cursor, + direction: flags.direction as 'next' | 'prev' | undefined, + }; + } case 'html': { if (!flags.url) { throw new Error('--url is required for the html sub-command.'); diff --git a/packages/@nitpicker/cli/src/query/types.ts b/packages/@nitpicker/cli/src/query/types.ts index 297db4c4..84f03ab6 100644 --- a/packages/@nitpicker/cli/src/query/types.ts +++ b/packages/@nitpicker/cli/src/query/types.ts @@ -5,6 +5,7 @@ export type QuerySubCommand = | 'summary' | 'pages' | 'page-detail' + | 'inbound-links' | 'html' | 'links' | 'resources' @@ -40,6 +41,7 @@ export const VALID_SUB_COMMANDS = [ 'summary', 'pages', 'page-detail', + 'inbound-links', 'html', 'links', 'resources', diff --git a/packages/@nitpicker/mcp-server/README.md b/packages/@nitpicker/mcp-server/README.md index 8b4a16a4..c536cdc1 100644 --- a/packages/@nitpicker/mcp-server/README.md +++ b/packages/@nitpicker/mcp-server/README.md @@ -23,19 +23,20 @@ stdio トランスポートで起動し、`.nitpicker` アーカイブを開い ## 主なツール -| ツール | 説明 | -| ------------------- | ----------------------------------------------------------------- | -| `open_archive` | `.nitpicker` ファイルを開き、archiveIdを返す | -| `close_archive` | アーカイブを閉じる | -| `get_summary` | サイト全体の概要統計 | -| `list_pages` | ページ一覧 | -| `get_page_detail` | 指定ページの詳細 | -| `get_page_html` | HTMLスナップショット | -| `list_links` | リンク一覧 | -| `list_resources` | リソース一覧 | -| `list_images` | 画像一覧 | -| `get_violations` | 分析プラグインの違反結果 | -| `list_console_logs` | 捕捉したconsoleログ・ページエラー（内容ごとに全ページ横断で集約） | +| ツール | 説明 | +| -------------------- | ----------------------------------------------------------------- | +| `open_archive` | `.nitpicker` ファイルを開き、archiveIdを返す | +| `close_archive` | アーカイブを閉じる | +| `get_summary` | サイト全体の概要統計 | +| `list_pages` | ページ一覧 | +| `get_page_detail` | 指定ページの詳細 | +| `list_inbound_links` | 指定ページへの被リンク一覧 | +| `get_page_html` | HTMLスナップショット | +| `list_links` | リンク一覧 | +| `list_resources` | リソース一覧 | +| `list_images` | 画像一覧 | +| `get_violations` | 分析プラグインの違反結果 | +| `list_console_logs` | 捕捉したconsoleログ・ページエラー（内容ごとに全ページ横断で集約） | ## 関連リンク diff --git a/packages/@nitpicker/mcp-server/src/mcp-server.spec.ts b/packages/@nitpicker/mcp-server/src/mcp-server.spec.ts index ee74b03f..149ad92e 100644 --- a/packages/@nitpicker/mcp-server/src/mcp-server.spec.ts +++ b/packages/@nitpicker/mcp-server/src/mcp-server.spec.ts @@ -2,6 +2,7 @@ import path from 'node:path'; import { tryParseUrl as parseUrl } from '@d-zero/shared/parse-url'; import { Archive } from '@nitpicker/crawler'; +import { buildViewerReadModel } from '@nitpicker/query'; import { afterAll, beforeAll, describe, expect, it } from 'vitest'; import { createServer } from './mcp-server.js'; @@ -253,6 +254,8 @@ describe('createServer', () => { ], ); + await buildViewerReadModel(archive); + await archive.write(); await archive.close(); @@ -264,11 +267,12 @@ describe('createServer', () => { rmSync(workingDir, { recursive: true, force: true }); }); - it('ListTools で31個のツールが返される', async () => { + it('ListTools で32個のツールが返される', async () => { const result = await listTools(server); - expect(result.tools).toHaveLength(31); + expect(result.tools).toHaveLength(32); const names = result.tools.map((t) => t.name); expect(names).toContain('open_archive'); + expect(names).toContain('list_inbound_links'); expect(names).toContain('close_archive'); expect(names).toContain('get_summary'); expect(names).toContain('list_isolated_clusters'); @@ -397,8 +401,6 @@ describe('createServer', () => { expect(data.title).toBe('Home'); expect(data.outboundLinks).toBeDefined(); expect(data.outboundLinks.length).toBe(1); - expect(data.inboundLinks).toBeDefined(); - expect(data.inboundLinks.length).toBe(0); }); it('get_page_detail で存在しないページは "Page not found." を返す', async () => { @@ -409,6 +411,40 @@ describe('createServer', () => { expect(result.content[0]!.text).toBe('Page not found.'); }); + it('list_inbound_links でページへの被リンクを取得する', async () => { + const result = await callTool(server, 'list_inbound_links', { + archiveId, + url: 'https://example.com/about', + }); + expect(result.isError).toBeUndefined(); + const data = JSON.parse(result.content[0]!.text); + expect(data.total).toBe(1); + expect(data.items).toHaveLength(1); + expect(data.items[0].url).toBe('https://example.com'); + expect(data.items[0].textContent).toBe('About us'); + expect(data.items[0].count).toBe(1); + }); + + it('list_inbound_links は limit: 0 で件数のみ返す', async () => { + const result = await callTool(server, 'list_inbound_links', { + archiveId, + url: 'https://example.com/about', + limit: 0, + }); + expect(result.isError).toBeUndefined(); + const data = JSON.parse(result.content[0]!.text); + expect(data.total).toBe(1); + expect(data.items).toHaveLength(0); + }); + + it('list_inbound_links で存在しないページは "Page not found." を返す', async () => { + const result = await callTool(server, 'list_inbound_links', { + archiveId, + url: 'https://example.com/nonexistent', + }); + expect(result.content[0]!.text).toBe('Page not found.'); + }); + it('get_page_html で HTML スナップショットを取得する', async () => { const result = await callTool(server, 'get_page_html', { archiveId, diff --git a/packages/@nitpicker/mcp-server/src/mcp-server.ts b/packages/@nitpicker/mcp-server/src/mcp-server.ts index e6347b0a..e23e488e 100644 --- a/packages/@nitpicker/mcp-server/src/mcp-server.ts +++ b/packages/@nitpicker/mcp-server/src/mcp-server.ts @@ -26,6 +26,7 @@ import { getTagInventory, getViolations, listConsoleLogs, + listInboundLinks, listIsolatedClustersFastPath, listIsolatedPagesFastPath, listLinks, @@ -256,6 +257,19 @@ export function createServer() { } return jsonResult(result); } + case 'list_inbound_links': { + const accessor = manager.get(requireString(args, 'archiveId')); + const url = requireString(args, 'url'); + const result = await listInboundLinks(accessor, { + url, + limit: optionalNumber(args, 'limit'), + cursor: optionalString(args, 'cursor'), + }); + if (!result) { + return textResult('Page not found.'); + } + return jsonResult(result); + } case 'get_page_html': { const accessor = manager.get(requireString(args, 'archiveId')); const url = requireString(args, 'url'); diff --git a/packages/@nitpicker/mcp-server/src/tool-definitions.ts b/packages/@nitpicker/mcp-server/src/tool-definitions.ts index d6217134..8071b889 100644 --- a/packages/@nitpicker/mcp-server/src/tool-definitions.ts +++ b/packages/@nitpicker/mcp-server/src/tool-definitions.ts @@ -146,7 +146,7 @@ export const toolDefinitions: Tool[] = [ { name: 'get_page_detail', description: - 'Get full details for a specific page URL: ~47 flat meta fields (title, description, OG, Twitter, robots, link, charset, manifest, themeColor, fb_app_id, verification_google, format_detection, og:image:alt/width/height, og:locale, og:article timestamps, twitter:site/creator, etc.), `metaExtras` JSON (referrer, viewport parsed, httpEquiv, apple, msapplication, verification.{bing|yandex|...}, geo, citation, hreflang alternates, others.*, originTrial), JSON-LD/SpeculationRules **summary** (count + unique @types + parseErrorCount), Wappalyzer tag **summary** (count + provider→ids map), main-content **aggregate counts only** (mainContentSelector, mainContentWordCount/BodyWordCount, mainContentHeadingCount/ImageCount/TableCount/ButtonCount/IframeCount/VideoCount/AudioCount/CanvasCount, scrollHeightDesktop/Mobile — null when the page was never rendered), outbound/inbound links, redirect sources, response headers, and within-archive timestamps (firstCrawledAt / lastCrawledAt). Raw JSON-LD entries, full tag rows, and the main-content child-entity arrays are NOT included — fetch them via `get_page_jsonld` / `get_page_tags` / `get_page_main_contents`. Use when drilling down into a specific page.', + "Get full details for a specific page URL: ~47 flat meta fields (title, description, OG, Twitter, robots, link, charset, manifest, themeColor, fb_app_id, verification_google, format_detection, og:image:alt/width/height, og:locale, og:article timestamps, twitter:site/creator, etc.), `metaExtras` JSON (referrer, viewport parsed, httpEquiv, apple, msapplication, verification.{bing|yandex|...}, geo, citation, hreflang alternates, others.*, originTrial), JSON-LD/SpeculationRules **summary** (count + unique @types + parseErrorCount), Wappalyzer tag **summary** (count + provider→ids map), main-content **aggregate counts only** (mainContentSelector, mainContentWordCount/BodyWordCount, mainContentHeadingCount/ImageCount/TableCount/ButtonCount/IframeCount/VideoCount/AudioCount/CanvasCount, scrollHeightDesktop/Mobile — null when the page was never rendered), outbound links, redirect sources, response headers, and within-archive timestamps (firstCrawledAt / lastCrawledAt). Inbound links are NOT included here — a page's referrer count can reach the hundreds of thousands on a large site; use `list_inbound_links` instead. Raw JSON-LD entries, full tag rows, and the main-content child-entity arrays are also NOT included — fetch them via `get_page_jsonld` / `get_page_tags` / `get_page_main_contents`. Use when drilling down into a specific page.", inputSchema: { type: 'object' as const, properties: { @@ -159,6 +159,33 @@ export const toolDefinitions: Tool[] = [ required: ['archiveId', 'url'], }, }, + { + name: 'list_inbound_links', + description: + "Find which pages link to a specific page — the reverse of the page's outbound links. Results are bounded and cursor-paginated (default 100 per call) — pass the returned `nextCursor` back in to fetch the rest for a page linked from many pages. Redirect sources and URL-normalization aliases resolve to their canonical page, the same resolution `get_page_detail` applies. Pass `limit: 0` to get only the total referrer count without any rows.", + inputSchema: { + type: 'object' as const, + properties: { + archiveId: { + type: 'string', + description: 'The archive ID returned by open_archive', + }, + url: { + type: 'string', + description: 'The exact URL of the page whose inbound links to list', + }, + limit: { + type: 'number', + description: 'Max referring pages (default: 100; 0 for count only)', + }, + cursor: { + type: 'string', + description: 'Opaque cursor from a previous call, taken from its `nextCursor`', + }, + }, + required: ['archiveId', 'url'], + }, + }, { name: 'get_page_html', description: diff --git a/packages/@nitpicker/query/src/get-page-detail.spec.ts b/packages/@nitpicker/query/src/get-page-detail.spec.ts index 9a829ddb..ff9d9b4b 100644 --- a/packages/@nitpicker/query/src/get-page-detail.spec.ts +++ b/packages/@nitpicker/query/src/get-page-detail.spec.ts @@ -101,38 +101,6 @@ describe('getPageDetail', () => { isSkipped: false, }); - // A referrer with two separate tags to the same target (e.g. a nav - // link and a footer link) — must count as one inbound link, not two. - await archive.setPage({ - url: parseUrl('https://example.com/duplicate-linker')!, - redirectPaths: [], - isExternal: false, - isTarget: true, - status: 200, - statusText: 'OK', - contentType: 'text/html', - contentLength: 300, - responseHeaders: {}, - html: '', - meta: makeBeholderMeta({ lang: 'ja', title: 'Duplicate linker' }), - anchorList: [ - { - href: parseUrl('https://example.com/about')!, - isExternal: false, - title: null, - textContent: 'About (nav)', - }, - { - href: parseUrl('https://example.com/about')!, - isExternal: false, - title: null, - textContent: 'About (footer)', - }, - ], - imageList: [], - isSkipped: false, - }); - await archive.setConsoleLogs( 'https://example.com/about', [], @@ -202,23 +170,6 @@ describe('getPageDetail', () => { expect(result!.outboundLinks[0]!.textContent).toBe('About us'); }); - it('インバウンドリンクを返す', async () => { - const result = await getPageDetail(archive, 'https://example.com/about'); - const inboundUrls = result!.inboundLinks.map((l) => l.url).toSorted(); - expect(inboundUrls).toEqual([ - 'https://example.com', - 'https://example.com/duplicate-linker', - ]); - }); - - it('同一ページから同じ宛先への複数アンカーは1件の被リンクに集約される', async () => { - const result = await getPageDetail(archive, 'https://example.com/about'); - const fromDuplicateLinker = result!.inboundLinks.filter( - (l) => l.url === 'https://example.com/duplicate-linker', - ); - expect(fromDuplicateLinker).toHaveLength(1); - }); - it('存在しないページは null を返す', async () => { const result = await getPageDetail(archive, 'https://example.com/nonexistent'); expect(result).toBeNull(); @@ -385,179 +336,6 @@ describe('getPageDetail: page_templates テーブル自体が存在しないア }); }); -describe('getPageDetail: 被リンクを redirect 越しに解決する（http/https 合算, #71）', () => { - let archive: InstanceType; - const dir = path.resolve(__dirname, '__test_fixtures_get_page_detail_redirect__'); - const archiveFilePath = path.resolve(dir, 'page-detail-redirect.nitpicker'); - - /** - * Minimal empty metadata object shared by the redirect-resolution fixtures. - * Avoids repeating the full nullable meta shape in every `setPage` call. - */ - const emptyMeta = { - lang: null, - title: null, - description: null, - keywords: null, - noindex: false, - nofollow: false, - noarchive: false, - canonical: null, - alternate: null, - 'og:type': null, - 'og:title': null, - 'og:site_name': null, - 'og:description': null, - 'og:url': null, - 'og:image': null, - 'twitter:card': null, - }; - - beforeAll(async () => { - const { mkdirSync } = await import('node:fs'); - mkdirSync(dir, { recursive: true }); - archive = await Archive.create({ filePath: archiveFilePath, cwd: dir }); - await archive.setConfig({ - baseUrl: 'https://example.com', - name: 'test', - version: '0.13.0', - recursive: true, - interval: 0, - image: true, - fetchExternal: false, - parallels: 1, - roots: ['https://example.com'], - excludes: [], - excludeKeywords: [], - excludeUrls: [], - maxExcludedDepth: 0, - retry: 3, - fromList: false, - disableQueries: false, - userAgent: 'test', - ignoreRobots: false, - }); - - // 1) Canonical destination — the https content page. - await archive.setPage({ - url: parseUrl('https://example.com/page')!, - redirectPaths: [], - isExternal: false, - isTarget: true, - status: 200, - statusText: 'OK', - contentType: 'text/html', - contentLength: 100, - responseHeaders: {}, - html: 'Page', - meta: { ...emptyMeta, title: 'Page' }, - anchorList: [], - imageList: [], - isSkipped: false, - }); - - // 2) http source that 301s to the https destination → http.redirectDestId = https.id. - await archive.setPage({ - url: parseUrl('http://example.com/page')!, - redirectPaths: ['https://example.com/page'], - isExternal: false, - isTarget: true, - status: 200, - statusText: 'OK', - contentType: 'text/html', - contentLength: 100, - responseHeaders: {}, - html: '', - meta: { ...emptyMeta }, - anchorList: [], - imageList: [], - isSkipped: false, - }); - - // 3) A page linking the https destination DIRECTLY. - await archive.setPage({ - url: parseUrl('https://example.com/linker-https')!, - redirectPaths: [], - isExternal: false, - isTarget: true, - status: 200, - statusText: 'OK', - contentType: 'text/html', - contentLength: 100, - responseHeaders: {}, - html: '', - meta: { ...emptyMeta }, - anchorList: [ - { - href: parseUrl('https://example.com/page')!, - isExternal: false, - title: null, - textContent: 'direct https', - }, - ], - imageList: [], - isSkipped: false, - }); - - // 4) A page linking the http SOURCE (which redirects to the https destination). - await archive.setPage({ - url: parseUrl('https://example.com/linker-http')!, - redirectPaths: [], - isExternal: false, - isTarget: true, - status: 200, - statusText: 'OK', - contentType: 'text/html', - contentLength: 100, - responseHeaders: {}, - html: '', - meta: { ...emptyMeta }, - anchorList: [ - { - href: parseUrl('http://example.com/page')!, - isExternal: false, - title: null, - textContent: 'via http', - }, - ], - imageList: [], - isSkipped: false, - }); - }); - - afterAll(async () => { - if (archive) { - await archive.close(); - } - const { rmSync } = await import('node:fs'); - rmSync(dir, { recursive: true, force: true }); - }); - - it('http リンクと https リンクが宛先ページの被リンクに合算される（分裂しない）', async () => { - const result = await getPageDetail(archive, 'https://example.com/page'); - const inboundUrls = result!.inboundLinks.map((l) => l.url).toSorted(); - // 直リンク(https) と redirect 元(http)へのリンク、両方が宛先に集約される。 - expect(inboundUrls).toEqual([ - 'https://example.com/linker-http', - 'https://example.com/linker-https', - ]); - }); - - it('redirect 元ページの URL で検索しても、宛先ページの詳細（被リンク込み）が返る', async () => { - // http://example.com/page (redirect 元) で検索した場合も、 - // https://example.com/page (宛先) で検索したのと同じ詳細に解決される — - // get-page-detail.ts の URL 解決が alias_of_id と同様に redirect_dest_id - // も辿るため。 - const result = await getPageDetail(archive, 'http://example.com/page'); - expect(result!.url).toBe('https://example.com/page'); - const inboundUrls = result!.inboundLinks.map((l) => l.url).toSorted(); - expect(inboundUrls).toEqual([ - 'https://example.com/linker-http', - 'https://example.com/linker-https', - ]); - }); -}); - describe('getPageDetail: content_items.alias_of_id handling', () => { let archive: InstanceType; const dir = path.resolve(__dirname, '__test_fixtures_get_page_detail_alias__'); @@ -621,32 +399,6 @@ describe('getPageDetail: content_items.alias_of_id handling', () => { imageList: [], isSkipped: false, }); - // Links directly at the alias URL — must resolve as an inbound link to - // the canonical page, same as redirect-source-targeted links do. - await archive.setPage({ - url: parseUrl('https://example.com/linker')!, - redirectPaths: [], - isExternal: false, - isTarget: true, - status: 200, - statusText: 'OK', - contentType: 'text/html', - contentLength: 100, - responseHeaders: {}, - html: '', - meta: makeBeholderMeta({ title: 'Linker' }), - anchorList: [ - { - href: parseUrl('https://example.com/index.html')!, - isExternal: false, - title: null, - textContent: 'Home via alias', - }, - ], - imageList: [], - isSkipped: false, - }); - const knex = archive.getKnex(); const target = await knex('content_items as ci') .join('url_refs as ur', 'ur.id', 'ci.url_id') @@ -681,12 +433,6 @@ describe('getPageDetail: content_items.alias_of_id handling', () => { expect(byAlias!.aliasUrls).toEqual(['https://example.com/index.html']); }); - it('resolves inbound links targeting the alias URL to the canonical page', async () => { - const result = await getPageDetail(archive, 'https://example.com'); - const inboundUrls = result!.inboundLinks.map((l) => l.url); - expect(inboundUrls).toEqual(['https://example.com/linker']); - }); - it('throws an actionable error when content_items.alias_of_id does not exist', async () => { const knex = archive.getKnex(); await knex.schema.alterTable('content_items', (t) => { diff --git a/packages/@nitpicker/query/src/get-page-detail.ts b/packages/@nitpicker/query/src/get-page-detail.ts index 411638c8..dd49d737 100644 --- a/packages/@nitpicker/query/src/get-page-detail.ts +++ b/packages/@nitpicker/query/src/get-page-detail.ts @@ -76,7 +76,7 @@ function summarizeTagRows(rows: readonly TagRow[]): PageDetail['tags'] { * @example * const detail = await getPageDetail(accessor, 'https://example.com/'); * if (detail) { - * console.log(detail.title, detail.status, detail.inboundLinks.length); + * console.log(detail.title, detail.status, detail.outboundLinks.length); * } */ export async function getPageDetail( @@ -287,33 +287,6 @@ export async function getPageDetail( }), ); - // Inbound links resolve through redirects: an anchor pointing at a - // redirect source is counted as an incoming link to the redirect's final - // destination. Grouped by referrer so a referrer with multiple - // anchor_edges to this page still yields exactly one row. - const inboundRows = (await knex('anchor_edges as ae') - .select('referrer_ur.url as url') - .min('text_ref.text as textContent') - .join('content_items as referrer', 'ae.page_id', 'referrer.id') - .join('url_refs as referrer_ur', 'referrer_ur.id', 'referrer.url_id') - .join('content_items as target', 'ae.href_page_id', 'target.id') - .leftJoin('text_refs as text_ref', 'text_ref.id', 'ae.first_text_id') - .whereRaw( - 'coalesce("target"."redirect_dest_id", "target"."alias_of_id", "target"."id") = ?', - [page.id], - ) - .groupBy('referrer.id', 'referrer_ur.url')) as { - url: string; - textContent: string | null; - }[]; - - const inboundLinks = inboundRows.map( - (row: { url: string; textContent: string | null }) => ({ - url: row.url, - textContent: row.textContent, - }), - ); - const redirectRows = await knex('content_items as ci') .join('url_refs as ur', 'ur.id', 'ci.url_id') .select('ur.url as url') @@ -422,7 +395,6 @@ export async function getPageDetail( tags: summarizeTagRows(tagRows), responseHeaders, outboundLinks, - inboundLinks, redirectFrom, aliasUrls, consoleLogs, diff --git a/packages/@nitpicker/query/src/list-inbound-links.spec.ts b/packages/@nitpicker/query/src/list-inbound-links.spec.ts new file mode 100644 index 00000000..019f2b41 --- /dev/null +++ b/packages/@nitpicker/query/src/list-inbound-links.spec.ts @@ -0,0 +1,802 @@ +import path from 'node:path'; + +import { tryParseUrl as parseUrl } from '@d-zero/shared/parse-url'; +import { Archive } from '@nitpicker/crawler'; +import { afterAll, beforeAll, describe, expect, it } from 'vitest'; + +import { listExternalLinks } from './list-external-links.js'; +import { listInboundLinks } from './list-inbound-links.js'; +import { makeBeholderMeta } from './test-helpers/make-beholder-meta.js'; +import { buildViewerReadModel } from './viewer-read-model/build-viewer-read-model.js'; + +const __filename = new URL(import.meta.url).pathname; +const __dirname = path.dirname(__filename); + +const BASE_CONFIG = { + baseUrl: 'https://example.com', + name: 'test', + version: '0.13.0', + recursive: true, + interval: 0, + image: true, + fetchExternal: false, + parallels: 1, + roots: ['https://example.com'], + excludes: [], + excludeKeywords: [], + excludeUrls: [], + maxExcludedDepth: 0, + retry: 3, + fromList: false, + disableQueries: false, + userAgent: 'test', + ignoreRobots: false, +}; + +describe('listInboundLinks', () => { + let archive: InstanceType; + const workingDir = path.resolve(__dirname, '__test_fixtures_list_inbound_links__'); + const archiveFilePath = path.resolve(workingDir, 'list-inbound-links-test.nitpicker'); + + beforeAll(async () => { + const { mkdirSync } = await import('node:fs'); + mkdirSync(workingDir, { recursive: true }); + archive = await Archive.create({ filePath: archiveFilePath, cwd: workingDir }); + await archive.setConfig(BASE_CONFIG); + + await archive.setPage({ + url: parseUrl('https://example.com/about')!, + redirectPaths: [], + isExternal: false, + isTarget: true, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 100, + responseHeaders: {}, + html: 'About', + meta: makeBeholderMeta({ title: 'About' }), + anchorList: [], + imageList: [], + isSkipped: false, + }); + await archive.setPage({ + url: parseUrl('https://example.com/referrer-a')!, + redirectPaths: [], + isExternal: false, + isTarget: true, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 100, + responseHeaders: {}, + html: '', + meta: makeBeholderMeta({ title: 'Referrer A' }), + anchorList: [ + { + href: parseUrl('https://example.com/about')!, + isExternal: false, + title: null, + textContent: 'About A', + }, + ], + imageList: [], + isSkipped: false, + }); + // Two separate tags to the same target (e.g. a nav link and a + // footer link) — must collapse to one inbound-link entry with count: 2, + // not two entries. + await archive.setPage({ + url: parseUrl('https://example.com/referrer-b')!, + redirectPaths: [], + isExternal: false, + isTarget: true, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 100, + responseHeaders: {}, + html: '', + meta: makeBeholderMeta({ title: 'Referrer B' }), + anchorList: [ + { + href: parseUrl('https://example.com/about')!, + isExternal: false, + title: null, + textContent: 'About B1', + }, + { + href: parseUrl('https://example.com/about')!, + isExternal: false, + title: null, + textContent: 'About B2', + }, + ], + imageList: [], + isSkipped: false, + }); + + // A referrer whose anchor carries no text (e.g. an image link with no + // alt) — must resolve to textContent: null, not an empty string or a + // thrown error. + await archive.setPage({ + url: parseUrl('https://example.com/referrer-c')!, + redirectPaths: [], + isExternal: false, + isTarget: true, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 100, + responseHeaders: {}, + html: '', + meta: makeBeholderMeta({ title: 'Referrer C' }), + anchorList: [ + { + href: parseUrl('https://example.com/about')!, + isExternal: false, + title: null, + textContent: null, + }, + ], + imageList: [], + isSkipped: false, + }); + + await buildViewerReadModel(archive); + }); + + afterAll(async () => { + if (archive) { + await archive.close(); + } + const { rmSync } = await import('node:fs'); + rmSync(workingDir, { recursive: true, force: true }); + }); + + it('lists referrers with anchor text and per-referrer count, one row per referrer', async () => { + const result = await listInboundLinks(archive, { url: 'https://example.com/about' }); + expect(result).not.toBeNull(); + expect(result!.total).toBe(3); + const byUrl = new Map(result!.items.map((item) => [item.url, item])); + expect(byUrl.get('https://example.com/referrer-a')).toEqual({ + url: 'https://example.com/referrer-a', + textContent: 'About A', + count: 1, + }); + // First-wins: the first tag's text ("About B1") is kept even + // though a second anchor to the same target follows it. + expect(byUrl.get('https://example.com/referrer-b')).toEqual({ + url: 'https://example.com/referrer-b', + textContent: 'About B1', + count: 2, + }); + // No anchor text — resolves to null, not '' or a thrown error. + expect(byUrl.get('https://example.com/referrer-c')).toEqual({ + url: 'https://example.com/referrer-c', + textContent: null, + count: 1, + }); + }); + + it('echoes back the requested url verbatim', async () => { + const result = await listInboundLinks(archive, { url: 'https://example.com/about' }); + expect(result!.url).toBe('https://example.com/about'); + }); + + it('returns null for a URL that does not match any page', async () => { + const result = await listInboundLinks(archive, { + url: 'https://example.com/nonexistent', + }); + expect(result).toBeNull(); + }); + + it('limit: 0 returns only the total, skipping the row window', async () => { + const result = await listInboundLinks(archive, { + url: 'https://example.com/about', + limit: 0, + }); + expect(result!.total).toBe(3); + expect(result!.items).toHaveLength(0); + }); + + it('limit: 0 never queries viewer_anchor_facts for rows, viewer_url_refs, or text_refs', async () => { + const knex = archive.getKnex(); + const queries: string[] = []; + const listener = (query: { sql: string }) => queries.push(query.sql); + knex.on('query', listener); + try { + await listInboundLinks(archive, { url: 'https://example.com/about', limit: 0 }); + } finally { + knex.removeListener('query', listener); + } + // The count query (`select count(*) ... from "viewer_anchor_facts"`) + // is expected and excluded here; anything selecting individual rows + // from viewer_anchor_facts, or resolving viewer_url_refs/text_refs + // (only needed once a row window exists), would mean the `limit: 0` + // short-circuit stopped short-circuiting. + const unexpectedQueries = queries.filter( + (sql) => + /from "viewer_url_refs"/i.test(sql) || + /from "text_refs"/i.test(sql) || + (/from "viewer_anchor_facts"/i.test(sql) && !/count/i.test(sql)), + ); + expect(unexpectedQueries).toEqual([]); + }); +}); + +describe('listInboundLinks: redirect resolution (http/https 合算, #71 の被リンク版)', () => { + let archive: InstanceType; + const workingDir = path.resolve( + __dirname, + '__test_fixtures_list_inbound_links_redirect__', + ); + const archiveFilePath = path.resolve( + workingDir, + 'list-inbound-links-redirect.nitpicker', + ); + + beforeAll(async () => { + const { mkdirSync } = await import('node:fs'); + mkdirSync(workingDir, { recursive: true }); + archive = await Archive.create({ filePath: archiveFilePath, cwd: workingDir }); + await archive.setConfig(BASE_CONFIG); + + // 1) Canonical destination — the https content page. + await archive.setPage({ + url: parseUrl('https://example.com/page')!, + redirectPaths: [], + isExternal: false, + isTarget: true, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 100, + responseHeaders: {}, + html: 'Page', + meta: makeBeholderMeta({ title: 'Page' }), + anchorList: [], + imageList: [], + isSkipped: false, + }); + + // 2) http source that 301s to the https destination. + await archive.setPage({ + url: parseUrl('http://example.com/page')!, + redirectPaths: ['https://example.com/page'], + isExternal: false, + isTarget: true, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 100, + responseHeaders: {}, + html: '', + meta: makeBeholderMeta(), + anchorList: [], + imageList: [], + isSkipped: false, + }); + + // 3) A page linking the https destination DIRECTLY. + await archive.setPage({ + url: parseUrl('https://example.com/linker-https')!, + redirectPaths: [], + isExternal: false, + isTarget: true, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 100, + responseHeaders: {}, + html: '', + meta: makeBeholderMeta(), + anchorList: [ + { + href: parseUrl('https://example.com/page')!, + isExternal: false, + title: null, + textContent: 'direct https', + }, + ], + imageList: [], + isSkipped: false, + }); + + // 4) A page linking the http SOURCE (which redirects to the https destination). + await archive.setPage({ + url: parseUrl('https://example.com/linker-http')!, + redirectPaths: [], + isExternal: false, + isTarget: true, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 100, + responseHeaders: {}, + html: '', + meta: makeBeholderMeta(), + anchorList: [ + { + href: parseUrl('http://example.com/page')!, + isExternal: false, + title: null, + textContent: 'via http', + }, + ], + imageList: [], + isSkipped: false, + }); + + await buildViewerReadModel(archive); + }); + + afterAll(async () => { + if (archive) { + await archive.close(); + } + const { rmSync } = await import('node:fs'); + rmSync(workingDir, { recursive: true, force: true }); + }); + + it('aggregates the direct https link and the link via the http redirect source together', async () => { + const result = await listInboundLinks(archive, { url: 'https://example.com/page' }); + const inboundUrls = result!.items.map((item) => item.url).toSorted(); + expect(inboundUrls).toEqual([ + 'https://example.com/linker-http', + 'https://example.com/linker-https', + ]); + }); + + it('resolves the same inbound links when queried by the redirect-source URL', async () => { + const result = await listInboundLinks(archive, { url: 'http://example.com/page' }); + const inboundUrls = result!.items.map((item) => item.url).toSorted(); + expect(inboundUrls).toEqual([ + 'https://example.com/linker-http', + 'https://example.com/linker-https', + ]); + }); +}); + +describe('listInboundLinks: content_items.alias_of_id handling', () => { + let archive: InstanceType; + const workingDir = path.resolve( + __dirname, + '__test_fixtures_list_inbound_links_alias__', + ); + const archiveFilePath = path.resolve(workingDir, 'list-inbound-links-alias.nitpicker'); + + beforeAll(async () => { + const { mkdirSync } = await import('node:fs'); + mkdirSync(workingDir, { recursive: true }); + archive = await Archive.create({ filePath: archiveFilePath, cwd: workingDir }); + await archive.setConfig(BASE_CONFIG); + + // Canonical `/` and its alias `/index.html`. + await archive.setPage({ + url: parseUrl('https://example.com/')!, + redirectPaths: [], + isExternal: false, + isTarget: true, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 100, + responseHeaders: {}, + html: 'Home', + meta: makeBeholderMeta({ title: 'Home' }), + anchorList: [], + imageList: [], + isSkipped: false, + }); + await archive.setPage({ + url: parseUrl('https://example.com/index.html')!, + redirectPaths: [], + isExternal: false, + isTarget: true, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 100, + responseHeaders: {}, + html: 'Home', + meta: makeBeholderMeta({ title: 'Home' }), + anchorList: [], + imageList: [], + isSkipped: false, + }); + // Links directly at the alias URL — must resolve as an inbound link to + // the canonical page, same as redirect-source-targeted links do. + await archive.setPage({ + url: parseUrl('https://example.com/linker')!, + redirectPaths: [], + isExternal: false, + isTarget: true, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 100, + responseHeaders: {}, + html: '', + meta: makeBeholderMeta({ title: 'Linker' }), + anchorList: [ + { + href: parseUrl('https://example.com/index.html')!, + isExternal: false, + title: null, + textContent: 'Home via alias', + }, + ], + imageList: [], + isSkipped: false, + }); + + const knex = archive.getKnex(); + const target = await knex('content_items as ci') + .join('url_refs as ur', 'ur.id', 'ci.url_id') + .select('ci.id as id') + .where('ur.url', 'https://example.com') + .first(); + const member = await knex('content_items as ci') + .join('url_refs as ur', 'ur.id', 'ci.url_id') + .select('ci.id as id') + .where('ur.url', 'https://example.com/index.html') + .first(); + await knex('content_items').where('id', member.id).update({ alias_of_id: target.id }); + + await buildViewerReadModel(archive); + }); + + afterAll(async () => { + if (archive) { + await archive.close(); + } + const { rmSync } = await import('node:fs'); + rmSync(workingDir, { recursive: true, force: true }); + }); + + it('resolves inbound links targeting the alias URL to the canonical page', async () => { + const result = await listInboundLinks(archive, { url: 'https://example.com' }); + const inboundUrls = result!.items.map((item) => item.url); + expect(inboundUrls).toEqual(['https://example.com/linker']); + }); + + it('resolves the same inbound links when queried by the alias URL itself', async () => { + const result = await listInboundLinks(archive, { + url: 'https://example.com/index.html', + }); + const inboundUrls = result!.items.map((item) => item.url); + expect(inboundUrls).toEqual(['https://example.com/linker']); + }); +}); + +describe('listInboundLinks: cursor pagination', () => { + let archive: InstanceType; + const workingDir = path.resolve( + __dirname, + '__test_fixtures_list_inbound_links_cursor__', + ); + const archiveFilePath = path.resolve(workingDir, 'list-inbound-links-cursor.nitpicker'); + + beforeAll(async () => { + const { mkdirSync } = await import('node:fs'); + mkdirSync(workingDir, { recursive: true }); + archive = await Archive.create({ filePath: archiveFilePath, cwd: workingDir }); + await archive.setConfig(BASE_CONFIG); + + await archive.setPage({ + url: parseUrl('https://example.com/target')!, + redirectPaths: [], + isExternal: false, + isTarget: true, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 100, + responseHeaders: {}, + html: '', + meta: makeBeholderMeta({ title: 'Target' }), + anchorList: [], + imageList: [], + isSkipped: false, + }); + for (const name of ['referrer-a', 'referrer-b']) { + await archive.setPage({ + url: parseUrl(`https://example.com/${name}`)!, + redirectPaths: [], + isExternal: false, + isTarget: true, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 100, + responseHeaders: {}, + html: '', + meta: makeBeholderMeta({ title: name }), + anchorList: [ + { + href: parseUrl('https://example.com/target')!, + isExternal: false, + title: null, + textContent: `Link from ${name}`, + }, + ], + imageList: [], + isSkipped: false, + }); + } + + await buildViewerReadModel(archive); + }); + + afterAll(async () => { + if (archive) { + await archive.close(); + } + const { rmSync } = await import('node:fs'); + rmSync(workingDir, { recursive: true, force: true }); + }); + + it('paginates forward via nextCursor with no duplicates or gaps', async () => { + const page1 = await listInboundLinks(archive, { + url: 'https://example.com/target', + limit: 1, + }); + expect(page1!.items).toHaveLength(1); + expect(page1!.total).toBe(2); + expect(page1!.nextCursor).not.toBeNull(); + expect(page1!.prevCursor).toBeNull(); + + const page2 = await listInboundLinks(archive, { + url: 'https://example.com/target', + limit: 1, + cursor: page1!.nextCursor!, + }); + expect(page2!.items).toHaveLength(1); + expect(page2!.total).toBe(2); + expect(page2!.nextCursor).toBeNull(); + expect(page2!.prevCursor).not.toBeNull(); + + expect([...page1!.items, ...page2!.items].map((item) => item.url).toSorted()).toEqual( + ['https://example.com/referrer-a', 'https://example.com/referrer-b'], + ); + }); + + it('walks backward from a forward cursor via direction: "prev" and restores the same page', async () => { + const page1 = await listInboundLinks(archive, { + url: 'https://example.com/target', + limit: 1, + }); + const page2 = await listInboundLinks(archive, { + url: 'https://example.com/target', + limit: 1, + cursor: page1!.nextCursor!, + }); + const back = await listInboundLinks(archive, { + url: 'https://example.com/target', + limit: 1, + cursor: page2!.prevCursor!, + direction: 'prev', + }); + expect(back!.items).toEqual(page1!.items); + }); + + it('supports a direct offset read for MPA page-number jumps', async () => { + const result = await listInboundLinks(archive, { + url: 'https://example.com/target', + limit: 1, + offset: 1, + }); + expect(result!.items).toHaveLength(1); + }); + + it('throws on a cursor minted for a different destPageId', async () => { + const page1 = await listInboundLinks(archive, { + url: 'https://example.com/target', + limit: 1, + }); + await archive.setPage({ + url: parseUrl('https://example.com/other-target')!, + redirectPaths: [], + isExternal: false, + isTarget: true, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 100, + responseHeaders: {}, + html: '', + meta: makeBeholderMeta({ title: 'Other target' }), + anchorList: [], + imageList: [], + isSkipped: false, + }); + await buildViewerReadModel(archive); + await expect( + listInboundLinks(archive, { + url: 'https://example.com/other-target', + limit: 1, + cursor: page1!.nextCursor!, + }), + ).rejects.toThrow(/does not match/); + }); +}); + +describe('listInboundLinks: viewer read model guard', () => { + let archive: InstanceType; + const workingDir = path.resolve( + __dirname, + '__test_fixtures_list_inbound_links_no_read_model__', + ); + const archiveFilePath = path.resolve( + workingDir, + 'list-inbound-links-no-read-model.nitpicker', + ); + + beforeAll(async () => { + const { mkdirSync } = await import('node:fs'); + mkdirSync(workingDir, { recursive: true }); + archive = await Archive.create({ filePath: archiveFilePath, cwd: workingDir }); + await archive.setConfig(BASE_CONFIG); + + await archive.setPage({ + url: parseUrl('https://example.com/about')!, + redirectPaths: [], + isExternal: false, + isTarget: true, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 100, + responseHeaders: {}, + html: '', + meta: makeBeholderMeta({ title: 'About' }), + anchorList: [], + imageList: [], + isSkipped: false, + }); + // No `buildViewerReadModel` call — the read model is never built. + }); + + afterAll(async () => { + if (archive) { + await archive.close(); + } + const { rmSync } = await import('node:fs'); + rmSync(workingDir, { recursive: true, force: true }); + }); + + it('throws an actionable error when the viewer read model is missing', async () => { + await expect( + listInboundLinks(archive, { url: 'https://example.com/about' }), + ).rejects.toThrow(/viewer-build/); + }); + + it('throws an actionable error when content_items.alias_of_id does not exist', async () => { + const knex = archive.getKnex(); + await knex.schema.alterTable('content_items', (t) => { + t.dropColumn('alias_of_id'); + }); + + await expect( + listInboundLinks(archive, { url: 'https://example.com/about' }), + ).rejects.toThrow(/viewer-build/); + + // Restore the column so afterAll's close()/other tests are unaffected. + await knex.schema.alterTable('content_items', (t) => { + t.integer('alias_of_id'); + }); + }); +}); + +describe('listInboundLinks: counting grain matches listExternalLinks.referrerCount', () => { + // Fixes ARCHITECTURE.md's invariant that both functions count referrers + // at the same grain (one row per referrer page, not per anchor) — a + // regression here would silently desync the External Links view's + // referrer count from Page Detail's / the inbound-links view's total. + let archive: InstanceType; + const workingDir = path.resolve( + __dirname, + '__test_fixtures_list_inbound_links_grain__', + ); + const archiveFilePath = path.resolve(workingDir, 'list-inbound-links-grain.nitpicker'); + + beforeAll(async () => { + const { mkdirSync } = await import('node:fs'); + mkdirSync(workingDir, { recursive: true }); + archive = await Archive.create({ filePath: archiveFilePath, cwd: workingDir }); + await archive.setConfig(BASE_CONFIG); + + await archive.setPage({ + url: parseUrl('https://example.com/referrer-a')!, + redirectPaths: [], + isExternal: false, + isTarget: true, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 100, + responseHeaders: {}, + html: '', + meta: makeBeholderMeta({ title: 'Referrer A' }), + anchorList: [ + { + href: parseUrl('https://external.example.com/')!, + isExternal: true, + title: null, + textContent: 'Ad banner', + }, + { + href: parseUrl('https://external.example.com/')!, + isExternal: true, + title: null, + textContent: 'Ad footer', + }, + ], + imageList: [], + isSkipped: false, + }); + await archive.setPage({ + url: parseUrl('https://example.com/referrer-b')!, + redirectPaths: [], + isExternal: false, + isTarget: true, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 100, + responseHeaders: {}, + html: '', + meta: makeBeholderMeta({ title: 'Referrer B' }), + anchorList: [ + { + href: parseUrl('https://external.example.com/')!, + isExternal: true, + title: null, + textContent: 'Ad sidebar', + }, + ], + imageList: [], + isSkipped: false, + }); + await archive.setPage({ + url: parseUrl('https://external.example.com/')!, + redirectPaths: [], + isExternal: true, + isTarget: false, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 100, + responseHeaders: {}, + html: '', + meta: makeBeholderMeta(), + anchorList: [], + imageList: [], + isSkipped: false, + }); + + await buildViewerReadModel(archive); + }); + + afterAll(async () => { + if (archive) { + await archive.close(); + } + const { rmSync } = await import('node:fs'); + rmSync(workingDir, { recursive: true, force: true }); + }); + + it('reports the same referrer count as listExternalLinks for the same destination', async () => { + const inbound = await listInboundLinks(archive, { + url: 'https://external.example.com', + }); + const external = await listExternalLinks(archive, {}); + const externalEntry = external.items.find( + (item) => item.destUrl === 'https://external.example.com', + ); + expect(externalEntry).toBeDefined(); + expect(inbound!.total).toBe(externalEntry!.referrerCount); + expect(inbound!.total).toBe(2); + }); +}); diff --git a/packages/@nitpicker/query/src/list-inbound-links.ts b/packages/@nitpicker/query/src/list-inbound-links.ts new file mode 100644 index 00000000..d84497dc --- /dev/null +++ b/packages/@nitpicker/query/src/list-inbound-links.ts @@ -0,0 +1,323 @@ +import type { + InboundLinkEntry, + InboundLinkList, + ListInboundLinksOptions, +} from './types.js'; +import type { + InboundLinksKeysetRow, + InboundLinksSortSpec, +} from './viewer-inbound-links-cursor/types.js'; +import type { ArchiveAccessor } from '@nitpicker/crawler'; +import type { Knex } from 'knex'; + +import { requireAliasOfIdColumn } from './require-alias-of-id-column.js'; +import { requireViewerReadModel } from './require-viewer-read-model.js'; +import { resolveAliasAndRedirectChain } from './resolve-alias-and-redirect-chain.js'; +import { readKeysetWindow } from './viewer-cursor-kit/read-keyset-window.js'; +import { buildInboundLinksFilterKey } from './viewer-inbound-links-cursor/build-inbound-links-filter-key.js'; +import { decodeInboundLinksCursor } from './viewer-inbound-links-cursor/decode-inbound-links-cursor.js'; +import { encodeInboundLinksCursor } from './viewer-inbound-links-cursor/encode-inbound-links-cursor.js'; +import { extractInboundLinksSortValues } from './viewer-inbound-links-cursor/extract-inbound-links-sort-values.js'; +import { getInboundLinksSortSpec } from './viewer-inbound-links-cursor/get-inbound-links-sort-spec.js'; +import { VIEWER_READ_MODEL_SCHEMA_VERSION } from './viewer-read-model/viewer-read-model-schema-version.js'; + +/** + * Constrains a `viewer_anchor_facts` query builder to one destination page's + * inbound edges — the only filter `listInboundLinks` supports. `dest_page_id` + * is index-covered by `vaf_dest(dest_page_id, edge_id)`, so both the count + * and the keyset window resolve in one index walk regardless of how many + * millions of rows the table holds overall. + * @param qb - The query builder to constrain. + * @param destPageId - The resolved canonical `content_items.id` to filter by. + */ +function applyInboundLinksFilters(qb: Knex.QueryBuilder, destPageId: number): void { + qb.where('dest_page_id', destPageId); +} + +/** + * Counts the total inbound edges for one destination page. + * @param knex - The archive's Knex instance. + * @param destPageId - The resolved canonical `content_items.id` to count. + * @returns The total referrer count. + */ +async function countInboundLinksTotal(knex: Knex, destPageId: number): Promise { + const qb = knex('viewer_anchor_facts'); + applyInboundLinksFilters(qb, destPageId); + const result = await qb.count<{ count: string }[]>({ count: '*' }); + return Number(result[0]?.count ?? 0); +} + +/** + * Runs one `viewer_anchor_facts` read via the shared {@link readKeysetWindow}, + * selecting the columns needed to resolve a referrer's URL and anchor text + * afterward. No join runs here: URL strings and anchor text are resolved + * from `viewer_url_refs`/`text_refs` only after the window is limit-bounded, + * the same deferred-resolution shape `listViewerBrokenLinks` uses. + * @param knex - The archive's Knex instance. + * @param destPageId - The resolved canonical `content_items.id` to filter by. + * @param spec - The resolved sort spec (columns to select/order by). + * @param orderDirection - The physical scan direction for this read. + * @param limit - The page size (the read fetches `limit + 1` rows). + * @param keyset - The keyset predicate to apply, or `undefined` for an + * unconstrained (initial / offset) read. + * @param offset - Row offset for a direct `OFFSET` read (page-number jumps). + * Ignored when `keyset` is supplied. + * @returns Up to `limit + 1` rows. + */ +async function readInboundLinksWindow( + knex: Knex, + destPageId: number, + spec: InboundLinksSortSpec, + orderDirection: 'asc' | 'desc', + limit: number, + keyset: { operator: '>' | '<'; values: readonly (string | number)[] } | undefined, + offset: number, +): Promise< + (InboundLinksKeysetRow & { + source_url_ref_id: number; + first_text_id: number | null; + count: number; + })[] +> { + return readKeysetWindow( + knex, + 'viewer_anchor_facts', + (qb) => applyInboundLinksFilters(qb, destPageId), + ['source_url_ref_id', 'first_text_id', 'count'], + spec, + orderDirection, + limit, + keyset, + offset, + ); +} + +/** + * Loads URL strings for the limited inbound-link window. + * @param knex - Query connection for the opened archive. + * @param refIds - `viewer_url_refs` ids selected by the keyset window. + * @returns A map from `viewer_url_refs.id` to URL. + */ +async function readUrlRefs( + knex: Knex, + refIds: readonly number[], +): Promise> { + if (refIds.length === 0) { + return new Map(); + } + const rows: { id: number; url: string }[] = await knex('viewer_url_refs') + .whereIn('id', [...new Set(refIds)]) + .select('id', 'url'); + return new Map(rows.map((row) => [row.id, row.url])); +} + +/** + * Loads anchor text for the limited inbound-link window. + * @param knex - Query connection for the opened archive. + * @param textIds - `text_refs` ids selected by the keyset window, `null` + * entries (anchors with no text) filtered out before the query. + * @returns A map from `text_refs.id` to anchor text. + */ +async function readTextRefs( + knex: Knex, + textIds: readonly (number | null)[], +): Promise> { + const ids = [...new Set(textIds.filter((id): id is number => id != null))]; + if (ids.length === 0) { + return new Map(); + } + const rows: { id: number; text: string }[] = await knex('text_refs') + .whereIn('id', ids) + .select('id', 'text'); + return new Map(rows.map((row) => [row.id, row.text])); +} + +/** + * Lists which pages link to a target page — the read-model-backed, + * cursor-paginated counterpart to `getPageDetail.outboundLinks`'s reverse + * direction, split out of `getPageDetail` itself because a page's referrer + * count can reach the hundreds of thousands on a large site (issue #235). + * + * Reads exclusively from `viewer_anchor_facts` (see `requireViewerReadModel`) + * — there is no legacy fallback, unlike most `viewer_*`-backed queries: + * answering "who links here" without the `dest_page_id` index would require + * a full `anchor_edges` scan resolved through + * `COALESCE(redirect_dest_id, alias_of_id, id)` for every row, the exact + * cost this function exists to avoid. Callers that must keep working during + * a live crawl (stub mode, where the read model cannot exist) must check + * `context.mode` and skip this query entirely rather than calling it. + * + * The initial read (no `cursor`), the forward keyset read, the backward + * keyset read, and the direct-`offset` read are four separate code paths, + * mirroring `listViewerBrokenLinks`/`listViewerDuplicateGroupPages`. + * @param accessor - The archive accessor to query. + * @param options - The target page's URL, plus pagination options. + * @returns A cursor-paginated list of inbound-link entries, or `null` if + * `options.url` does not match any page in the archive. + * @throws {Error} If `content_items.alias_of_id` does not exist on this + * connection (see `requireAliasOfIdColumn`), or if the viewer read model + * is missing or stale (see `requireViewerReadModel`). + * @throws {Error} If `options.cursor` is malformed, stale, or was minted + * under a different filter/sort combination. + * @example + * // Page Detail's referrer count — skip the row window entirely: + * const counted = await listInboundLinks(accessor, { url, limit: 0 }); + * console.log(counted?.total); + * @example + * // Virtual-scroll continuation — the caller only ever inspects nextCursor: + * const page1 = await listInboundLinks(accessor, { url, limit: 100 }); + * const page2 = page1?.nextCursor + * ? await listInboundLinks(accessor, { url, limit: 100, cursor: page1.nextCursor }) + * : null; + */ +export async function listInboundLinks( + accessor: ArchiveAccessor, + options: ListInboundLinksOptions, +): Promise { + const knex = accessor.getKnex(); + await requireAliasOfIdColumn(knex); + await requireViewerReadModel(accessor); + + const candidate = await knex('content_items as ci') + .join('url_refs as ur', 'ur.id', 'ci.url_id') + .select('ci.id as id') + .where('ur.url', options.url) + .first(); + if (!candidate) { + return null; + } + const destPageId = await resolveAliasAndRedirectChain(knex, candidate.id); + + const limit = options.limit ?? 100; + const spec = getInboundLinksSortSpec(); + const filterKey = buildInboundLinksFilterKey({ destPageId }); + + const total = await countInboundLinksTotal(knex, destPageId); + + if (limit === 0) { + return { + url: options.url, + items: [], + total, + limit: 0, + offset: options.offset ?? 0, + nextCursor: null, + prevCursor: null, + }; + } + + /** + * Builds the final result from a `limit`-or-fewer window, already in + * final display order. + * @param window - The trimmed row window. + * @param hasMoreAfter - Whether a subsequent page exists. + * @param hasMoreBefore - Whether a preceding page exists. + * @returns The full paginated result. + */ + async function buildResult( + window: Awaited>, + hasMoreAfter: boolean, + hasMoreBefore: boolean, + ): Promise { + const [urlByRefId, textByRefId] = await Promise.all([ + readUrlRefs( + knex, + window.map((row) => row.source_url_ref_id), + ), + readTextRefs( + knex, + window.map((row) => row.first_text_id), + ), + ]); + const items: InboundLinkEntry[] = window.map((row) => { + const url = urlByRefId.get(row.source_url_ref_id); + if (url == null) { + throw new Error( + `listInboundLinks: missing viewer_url_refs row ${row.source_url_ref_id}`, + ); + } + return { + url, + textContent: + row.first_text_id == null ? null : (textByRefId.get(row.first_text_id) ?? null), + count: row.count, + }; + }); + const lastRow = window.at(-1); + const firstRow = window[0]; + const nextCursor = + hasMoreAfter && lastRow + ? encodeInboundLinksCursor({ + v: VIEWER_READ_MODEL_SCHEMA_VERSION, + filterKey, + sortBy: 'edgeId', + sortOrder: 'asc', + values: extractInboundLinksSortValues(spec, lastRow), + }) + : null; + const prevCursor = + hasMoreBefore && firstRow + ? encodeInboundLinksCursor({ + v: VIEWER_READ_MODEL_SCHEMA_VERSION, + filterKey, + sortBy: 'edgeId', + sortOrder: 'asc', + values: extractInboundLinksSortValues(spec, firstRow), + }) + : null; + return { + url: options.url, + items, + total, + limit, + offset: options.offset ?? 0, + nextCursor, + prevCursor, + }; + } + + if (options.cursor) { + const decoded = decodeInboundLinksCursor(options.cursor, { filterKey }); + if (options.direction === 'prev') { + const oppositeDirection = spec.scanDirection === 'asc' ? 'desc' : 'asc'; + const fetched = await readInboundLinksWindow( + knex, + destPageId, + spec, + oppositeDirection, + limit, + { operator: spec.scanDirection === 'asc' ? '<' : '>', values: decoded.values }, + 0, + ); + const hasMoreBefore = fetched.length > limit; + const window = fetched.slice(0, limit).toReversed(); + return await buildResult(window, true, hasMoreBefore); + } + const fetched = await readInboundLinksWindow( + knex, + destPageId, + spec, + spec.scanDirection, + limit, + { operator: spec.scanDirection === 'asc' ? '>' : '<', values: decoded.values }, + 0, + ); + const hasMoreAfter = fetched.length > limit; + const window = fetched.slice(0, limit); + return await buildResult(window, hasMoreAfter, true); + } + + const offset = options.offset ?? 0; + const fetched = await readInboundLinksWindow( + knex, + destPageId, + spec, + spec.scanDirection, + limit, + undefined, + offset, + ); + const hasMoreAfter = fetched.length > limit; + const window = fetched.slice(0, limit); + return await buildResult(window, hasMoreAfter, offset > 0); +} diff --git a/packages/@nitpicker/query/src/query.ts b/packages/@nitpicker/query/src/query.ts index 3c331497..f1ee2c04 100644 --- a/packages/@nitpicker/query/src/query.ts +++ b/packages/@nitpicker/query/src/query.ts @@ -63,6 +63,7 @@ export { listDirectoryChildren } from './list-directory-children.js'; export { listDirectoryPages } from './list-directory-pages.js'; export { listExternalLinks } from './list-external-links.js'; export { listImages } from './list-images.js'; +export { listInboundLinks } from './list-inbound-links.js'; export { listInventoryRuns } from './list-inventory-runs.js'; export { listIsolatedClustersFastPath } from './list-isolated-clusters-fast-path.js'; export { listIsolatedClusters } from './list-isolated-clusters.js'; diff --git a/packages/@nitpicker/query/src/require-viewer-read-model.spec.ts b/packages/@nitpicker/query/src/require-viewer-read-model.spec.ts new file mode 100644 index 00000000..936cdca3 --- /dev/null +++ b/packages/@nitpicker/query/src/require-viewer-read-model.spec.ts @@ -0,0 +1,52 @@ +import path from 'node:path'; + +import { Archive } from '@nitpicker/crawler'; +import { afterAll, beforeAll, describe, expect, it } from 'vitest'; + +import { requireViewerReadModel } from './require-viewer-read-model.js'; +import { createViewerReadModelTables } from './viewer-read-model/create-viewer-read-model-tables.js'; +import { VIEWER_READ_MODEL_SCHEMA_VERSION } from './viewer-read-model/viewer-read-model-schema-version.js'; + +const __filename = new URL(import.meta.url).pathname; +const __dirname = path.dirname(__filename); +const workingDir = path.resolve(__dirname, '__test_fixtures_require_viewer_read_model__'); + +describe('requireViewerReadModel', () => { + let archive: InstanceType; + const archiveFilePath = path.resolve( + workingDir, + 'require-viewer-read-model-test.nitpicker', + ); + + beforeAll(async () => { + const { mkdirSync } = await import('node:fs'); + mkdirSync(workingDir, { recursive: true }); + archive = await Archive.create({ filePath: archiveFilePath, cwd: workingDir }); + }); + + afterAll(async () => { + if (archive) { + await archive.releaseHandle(); + } + const { rmSync } = await import('node:fs'); + rmSync(workingDir, { recursive: true, force: true }); + }); + + it('throws an actionable error when the read model has not been built', async () => { + await expect(requireViewerReadModel(archive)).rejects.toThrow(/viewer-build/); + }); + + it('resolves once the read model is at the current schema version', async () => { + const knex = archive.getKnex(); + await knex.transaction(async (trx) => { + await createViewerReadModelTables(trx); + await trx('viewer_read_model_meta').insert({ + id: 1, + schema_version: VIEWER_READ_MODEL_SCHEMA_VERSION, + built_at: 1, + source_row_count: 0, + }); + }); + await expect(requireViewerReadModel(archive)).resolves.toBeUndefined(); + }); +}); diff --git a/packages/@nitpicker/query/src/require-viewer-read-model.ts b/packages/@nitpicker/query/src/require-viewer-read-model.ts new file mode 100644 index 00000000..32b50ac3 --- /dev/null +++ b/packages/@nitpicker/query/src/require-viewer-read-model.ts @@ -0,0 +1,36 @@ +import type { ArchiveAccessor } from '@nitpicker/crawler'; + +import { isViewerReadModelCurrent } from './viewer-read-model/is-viewer-read-model-current.js'; + +/** + * Throws an actionable error when the viewer read model is missing or + * stale on this accessor, instead of letting a query that depends on it + * (e.g. reading `viewer_anchor_facts`) fail with a raw `no such table` + * error or silently read a stale snapshot. + * + * Unlike `isViewerReadModelCurrent`'s other callers (`get-*-fast-path.ts`), + * which branch to a legacy SQL fallback when the read model isn't current, + * `listInboundLinks` has no legacy fallback (see its docs, #235) — the + * `viewer_anchor_facts.dest_page_id` index is the only way to answer + * "who links here" without a full `anchor_edges` scan, so there is nothing + * correct to fall back to. This mirrors `requireAliasOfIdColumn`'s pattern + * of failing closed with a `viewer-build` pointer rather than branching. + * + * Never call this from a route/query that must keep working during a live + * crawl (stub mode) — the read model cannot exist there (`buildViewerReadModel` + * refuses read-only accessors, and `viewer-build` refuses stub directories), + * so this always throws in stub mode. Callers needing stub-mode compatibility + * must check `context.mode` and skip the read-model-dependent query entirely + * instead of calling this guard. + * @param accessor - The archive accessor to check. + * @throws {Error} If the viewer read model is missing or built under a + * stale schema version. + */ +export async function requireViewerReadModel(accessor: ArchiveAccessor): Promise { + if (!(await isViewerReadModelCurrent(accessor))) { + throw new Error( + "This archive's viewer read model is missing or stale. " + + 'Run `viewer-build` against it once to build it first.', + ); + } +} diff --git a/packages/@nitpicker/query/src/types.ts b/packages/@nitpicker/query/src/types.ts index 04440805..5ee21dab 100644 --- a/packages/@nitpicker/query/src/types.ts +++ b/packages/@nitpicker/query/src/types.ts @@ -1237,8 +1237,11 @@ export interface PageDetail { responseHeaders: Record; /** Outgoing links from this page. */ outboundLinks: OutboundLink[]; - /** Incoming links to this page. */ - inboundLinks: InboundLink[]; + // Inbound links are deliberately not embedded here: a page's referrer + // count can reach the hundreds of thousands on large sites (unlike + // outbound links, which are bounded by that one page's own anchors), so + // they are read separately via `listInboundLinks`, a bounded, + // cursor-paginated query (#235). /** URLs that redirect to this page. */ redirectFrom: string[]; /** URLs merged into this page via URL-normalization equivalence (`content_items.alias_of_id`). */ @@ -1550,13 +1553,80 @@ export interface OutboundLink { } /** - * An incoming link pointing to a page. + * Options for {@link import('./list-inbound-links.js').listInboundLinks}. */ -export interface InboundLink { +export interface ListInboundLinksOptions { + /** + * The exact URL of the page whose inbound links to list. A redirect + * source or URL-normalization alias resolves to its canonical page's + * inbound links, the same resolution `getPageDetail` applies. + */ + url: string; + /** + * Maximum number of results to return. Defaults to 100. Pass `0` to skip + * the row window entirely and return only `total` with empty `items` — + * the shape a caller that only needs a count (e.g. Page Detail's + * referrer count) should request. + */ + limit?: number; + /** + * Opaque keyset cursor from a previous {@link InboundLinkList}'s + * `nextCursor`/`prevCursor`. Mutually exclusive with `offset` — when both + * are supplied, `cursor` wins. Omit for the first page. + */ + cursor?: string; + /** + * Direction to walk from `cursor`: `'next'` (forward, default) or + * `'prev'` (backward). Ignored when `cursor` is omitted. + */ + direction?: 'next' | 'prev'; + /** + * Row offset for page-number jumps (MPA pagination). Mutually exclusive + * with `cursor`. + */ + offset?: number; +} + +/** + * One inbound link — a distinct referrer page linking to the target, with + * its first-observed anchor text and the number of anchors on that referrer + * pointing here (see `anchor_edges`'s first-wins dedup — this is the same + * grain: one row per referrer, not per anchor). + */ +export interface InboundLinkEntry { /** The URL of the referring page. */ url: string; - /** The anchor text content. */ + /** The first-observed anchor text content, or `null` if it carried none. */ textContent: string | null; + /** Number of anchors on the referrer page pointing to the target. */ + count: number; +} + +/** + * Result of {@link import('./list-inbound-links.js').listInboundLinks} — a + * bounded, cursor-paginated window of pages linking to a target page. + */ +export interface InboundLinkList { + /** The target page's URL, as supplied in the request. */ + url: string; + /** Inbound link entries in this window, bounded to at most `limit`. */ + items: InboundLinkEntry[]; + /** Total distinct referrer pages linking to the target — independent of `items`' window length. */ + total: number; + /** Current limit. */ + limit: number; + /** Current offset. */ + offset: number; + /** + * Opaque cursor to fetch the next page in the current sort order, or + * `null` when this is the last page. + */ + nextCursor: string | null; + /** + * Opaque cursor to fetch the preceding page, or `null` when this is + * already the first page. + */ + prevCursor: string | null; } /** diff --git a/packages/@nitpicker/query/src/viewer-inbound-links-cursor/build-inbound-links-filter-key.spec.ts b/packages/@nitpicker/query/src/viewer-inbound-links-cursor/build-inbound-links-filter-key.spec.ts new file mode 100644 index 00000000..3dcdc490 --- /dev/null +++ b/packages/@nitpicker/query/src/viewer-inbound-links-cursor/build-inbound-links-filter-key.spec.ts @@ -0,0 +1,21 @@ +import { describe, expect, it } from 'vitest'; + +import { buildInboundLinksFilterKey } from './build-inbound-links-filter-key.js'; + +describe('buildInboundLinksFilterKey', () => { + it('produces the canonical {"destPageId":N} JSON shape', () => { + expect(buildInboundLinksFilterKey({ destPageId: 1 })).toBe('{"destPageId":1}'); + }); + + it('produces different keys for different destPageIds', () => { + expect(buildInboundLinksFilterKey({ destPageId: 1 })).not.toBe( + buildInboundLinksFilterKey({ destPageId: 2 }), + ); + }); + + it('produces the same key for the same destPageId across calls', () => { + expect(buildInboundLinksFilterKey({ destPageId: 1 })).toBe( + buildInboundLinksFilterKey({ destPageId: 1 }), + ); + }); +}); diff --git a/packages/@nitpicker/query/src/viewer-inbound-links-cursor/build-inbound-links-filter-key.ts b/packages/@nitpicker/query/src/viewer-inbound-links-cursor/build-inbound-links-filter-key.ts new file mode 100644 index 00000000..255e634b --- /dev/null +++ b/packages/@nitpicker/query/src/viewer-inbound-links-cursor/build-inbound-links-filter-key.ts @@ -0,0 +1,15 @@ +import type { InboundLinksCursorFilterKeyInput } from './types.js'; + +import { buildFilterKey } from '../viewer-cursor-kit/build-filter-key.js'; + +/** + * Builds the normalized `filterKey` embedded in a cursor. Thin wrapper over + * the shared {@link buildFilterKey}. + * @param filters - The filter-affecting subset of the caller's options. + * @returns A canonical JSON string uniquely identifying the filter set. + */ +export function buildInboundLinksFilterKey( + filters: InboundLinksCursorFilterKeyInput, +): string { + return buildFilterKey({ destPageId: filters.destPageId }); +} diff --git a/packages/@nitpicker/query/src/viewer-inbound-links-cursor/decode-inbound-links-cursor.spec.ts b/packages/@nitpicker/query/src/viewer-inbound-links-cursor/decode-inbound-links-cursor.spec.ts new file mode 100644 index 00000000..c4a6c276 --- /dev/null +++ b/packages/@nitpicker/query/src/viewer-inbound-links-cursor/decode-inbound-links-cursor.spec.ts @@ -0,0 +1,65 @@ +import { describe, expect, it } from 'vitest'; + +import { VIEWER_READ_MODEL_SCHEMA_VERSION } from '../viewer-read-model/viewer-read-model-schema-version.js'; + +import { decodeInboundLinksCursor } from './decode-inbound-links-cursor.js'; +import { encodeInboundLinksCursor } from './encode-inbound-links-cursor.js'; + +const PAYLOAD_BASE = { + filterKey: '{"destPageId":1}', + sortBy: 'edgeId' as const, + sortOrder: 'asc' as const, +}; + +const EXPECTED = { filterKey: PAYLOAD_BASE.filterKey }; + +describe('decodeInboundLinksCursor', () => { + it('decodes a cursor that matches the expected filter', () => { + const cursor = encodeInboundLinksCursor({ + v: VIEWER_READ_MODEL_SCHEMA_VERSION, + ...PAYLOAD_BASE, + values: [7], + }); + expect(decodeInboundLinksCursor(cursor, EXPECTED)).toEqual({ + v: VIEWER_READ_MODEL_SCHEMA_VERSION, + ...PAYLOAD_BASE, + values: [7], + }); + }); + + it('throws on an undecodable string', () => { + expect(() => decodeInboundLinksCursor('%%%not-base64%%%', EXPECTED)).toThrow( + /not decodable/, + ); + }); + + it('throws on a cursor minted under a stale schema version', () => { + const cursor = encodeInboundLinksCursor({ + v: VIEWER_READ_MODEL_SCHEMA_VERSION - 1, + ...PAYLOAD_BASE, + values: [7], + }); + expect(() => decodeInboundLinksCursor(cursor, EXPECTED)).toThrow(/[Ss]tale/); + }); + + it('throws on a cursor minted for a different destPageId', () => { + const cursor = encodeInboundLinksCursor({ + v: VIEWER_READ_MODEL_SCHEMA_VERSION, + ...PAYLOAD_BASE, + filterKey: '{"destPageId":2}', + values: [7], + }); + expect(() => decodeInboundLinksCursor(cursor, EXPECTED)).toThrow(/does not match/); + }); + + it('throws on a values array whose length does not match the expected column count', () => { + const cursor = encodeInboundLinksCursor({ + v: VIEWER_READ_MODEL_SCHEMA_VERSION, + ...PAYLOAD_BASE, + values: [7, 8], + }); + expect(() => decodeInboundLinksCursor(cursor, EXPECTED)).toThrow( + /keyset value count/, + ); + }); +}); diff --git a/packages/@nitpicker/query/src/viewer-inbound-links-cursor/decode-inbound-links-cursor.ts b/packages/@nitpicker/query/src/viewer-inbound-links-cursor/decode-inbound-links-cursor.ts new file mode 100644 index 00000000..bd9d6de3 --- /dev/null +++ b/packages/@nitpicker/query/src/viewer-inbound-links-cursor/decode-inbound-links-cursor.ts @@ -0,0 +1,39 @@ +import type { InboundLinksCursorPayload } from './types.js'; + +import { decodeCursorEnvelope } from '../viewer-cursor-kit/decode-cursor-envelope.js'; + +/** + * The current request's identity to validate a decoded cursor against. + */ +export interface ExpectedInboundLinksCursor { + /** See `buildInboundLinksFilterKey`. */ + filterKey: string; +} + +/** + * Decodes and validates an opaque cursor against the caller's current + * `destPageId`. Rejects cursors minted under a different schema version or a + * different `destPageId` — replaying a cursor minted for one page's inbound + * links against another's would silently seek to a nonsensical position. + * Thin wrapper over the shared {@link decodeCursorEnvelope}. + * @param cursor - The opaque cursor string from the request. + * @param expected - The current request's filter key, to validate against. + * @returns The decoded, validated payload. + * @throws {Error} If the cursor is malformed, stale, or was minted under a + * different `destPageId`. + */ +export function decodeInboundLinksCursor( + cursor: string, + expected: ExpectedInboundLinksCursor, +): InboundLinksCursorPayload { + return decodeCursorEnvelope( + cursor, + { + filterKey: expected.filterKey, + sortBy: 'edgeId', + sortOrder: 'asc', + expectedValueCount: 1, + }, + '/api/pages/inbound-links', + ); +} diff --git a/packages/@nitpicker/query/src/viewer-inbound-links-cursor/encode-inbound-links-cursor.spec.ts b/packages/@nitpicker/query/src/viewer-inbound-links-cursor/encode-inbound-links-cursor.spec.ts new file mode 100644 index 00000000..63deb39a --- /dev/null +++ b/packages/@nitpicker/query/src/viewer-inbound-links-cursor/encode-inbound-links-cursor.spec.ts @@ -0,0 +1,19 @@ +import { describe, expect, it } from 'vitest'; + +import { encodeInboundLinksCursor } from './encode-inbound-links-cursor.js'; + +describe('encodeInboundLinksCursor', () => { + it('round-trips through base64url without loss', () => { + const payload = { + v: 22, + filterKey: '{"destPageId":1}', + sortBy: 'edgeId' as const, + sortOrder: 'asc' as const, + values: [7], + }; + const cursor = encodeInboundLinksCursor(payload); + expect(JSON.parse(Buffer.from(cursor, 'base64url').toString('utf8'))).toEqual( + payload, + ); + }); +}); diff --git a/packages/@nitpicker/query/src/viewer-inbound-links-cursor/encode-inbound-links-cursor.ts b/packages/@nitpicker/query/src/viewer-inbound-links-cursor/encode-inbound-links-cursor.ts new file mode 100644 index 00000000..49433fe0 --- /dev/null +++ b/packages/@nitpicker/query/src/viewer-inbound-links-cursor/encode-inbound-links-cursor.ts @@ -0,0 +1,13 @@ +import type { InboundLinksCursorPayload } from './types.js'; + +import { encodeCursorEnvelope } from '../viewer-cursor-kit/encode-cursor-envelope.js'; + +/** + * Encodes a cursor payload as an opaque, URL-safe string. Thin wrapper over + * the shared {@link encodeCursorEnvelope}. + * @param payload - The cursor payload to encode. + * @returns The base64url-encoded cursor. + */ +export function encodeInboundLinksCursor(payload: InboundLinksCursorPayload): string { + return encodeCursorEnvelope(payload); +} diff --git a/packages/@nitpicker/query/src/viewer-inbound-links-cursor/extract-inbound-links-sort-values.spec.ts b/packages/@nitpicker/query/src/viewer-inbound-links-cursor/extract-inbound-links-sort-values.spec.ts new file mode 100644 index 00000000..b3466ddb --- /dev/null +++ b/packages/@nitpicker/query/src/viewer-inbound-links-cursor/extract-inbound-links-sort-values.spec.ts @@ -0,0 +1,16 @@ +import type { InboundLinksKeysetRow } from './types.js'; + +import { describe, expect, it } from 'vitest'; + +import { extractInboundLinksSortValues } from './extract-inbound-links-sort-values.js'; + +describe('extractInboundLinksSortValues', () => { + it('extracts values in spec.columns order', () => { + const row: InboundLinksKeysetRow = { edge_id: 7 }; + const values = extractInboundLinksSortValues( + { columns: ['edge_id'], scanDirection: 'asc' }, + row, + ); + expect(values).toEqual([7]); + }); +}); diff --git a/packages/@nitpicker/query/src/viewer-inbound-links-cursor/extract-inbound-links-sort-values.ts b/packages/@nitpicker/query/src/viewer-inbound-links-cursor/extract-inbound-links-sort-values.ts new file mode 100644 index 00000000..51a83819 --- /dev/null +++ b/packages/@nitpicker/query/src/viewer-inbound-links-cursor/extract-inbound-links-sort-values.ts @@ -0,0 +1,18 @@ +import type { InboundLinksKeysetRow, InboundLinksSortSpec } from './types.js'; + +import { extractKeysetSortValues } from '../viewer-cursor-kit/extract-keyset-sort-values.js'; + +/** + * Extracts a row's keyset tuple values in `spec.columns` order — the values + * bound into a cursor's comparison tuple. Thin wrapper over the shared + * {@link extractKeysetSortValues}. + * @param spec - The sort spec whose columns to read. + * @param row - The source row (must carry every column in `spec.columns`). + * @returns The tuple values, in `spec.columns` order. + */ +export function extractInboundLinksSortValues( + spec: InboundLinksSortSpec, + row: InboundLinksKeysetRow, +): (string | number)[] { + return extractKeysetSortValues(spec, row); +} diff --git a/packages/@nitpicker/query/src/viewer-inbound-links-cursor/get-inbound-links-sort-spec.spec.ts b/packages/@nitpicker/query/src/viewer-inbound-links-cursor/get-inbound-links-sort-spec.spec.ts new file mode 100644 index 00000000..450d31d7 --- /dev/null +++ b/packages/@nitpicker/query/src/viewer-inbound-links-cursor/get-inbound-links-sort-spec.spec.ts @@ -0,0 +1,12 @@ +import { describe, expect, it } from 'vitest'; + +import { getInboundLinksSortSpec } from './get-inbound-links-sort-spec.js'; + +describe('getInboundLinksSortSpec', () => { + it('always sorts by edge_id ascending', () => { + expect(getInboundLinksSortSpec()).toEqual({ + columns: ['edge_id'], + scanDirection: 'asc', + }); + }); +}); diff --git a/packages/@nitpicker/query/src/viewer-inbound-links-cursor/get-inbound-links-sort-spec.ts b/packages/@nitpicker/query/src/viewer-inbound-links-cursor/get-inbound-links-sort-spec.ts new file mode 100644 index 00000000..120c2b8f --- /dev/null +++ b/packages/@nitpicker/query/src/viewer-inbound-links-cursor/get-inbound-links-sort-spec.ts @@ -0,0 +1,12 @@ +import type { InboundLinksSortSpec } from './types.js'; + +/** + * Resolves the keyset sort plan for `viewer_anchor_facts` when read by + * `dest_page_id`. There is only one supported order (`edge_id` ascending) — + * see {@link InboundLinksSortSpec}'s docs for why no `sortBy`/`sortOrder` + * parameter is needed. + * @returns The fixed {@link InboundLinksSortSpec}. + */ +export function getInboundLinksSortSpec(): InboundLinksSortSpec { + return { columns: ['edge_id'], scanDirection: 'asc' }; +} diff --git a/packages/@nitpicker/query/src/viewer-inbound-links-cursor/types.ts b/packages/@nitpicker/query/src/viewer-inbound-links-cursor/types.ts new file mode 100644 index 00000000..a48b7f3e --- /dev/null +++ b/packages/@nitpicker/query/src/viewer-inbound-links-cursor/types.ts @@ -0,0 +1,54 @@ +/** + * The only keyset tuple this cursor family supports: `(edge_id)`. + * `viewer_anchor_facts`'s `vaf_dest(dest_page_id, edge_id)` index already + * covers both the `dest_page_id = ?` filter and an `edge_id` order/seek in + * one index walk, so there is no `sortBy`/`sortOrder`-dependent column + * switch here — the same "one fixed order" shape + * `viewer-duplicate-group-pages-cursor` uses. + */ +export interface InboundLinksSortSpec { + /** Keyset tuple columns, in comparison/`ORDER BY` order. */ + readonly columns: readonly ['edge_id']; + /** Physical scan direction that yields display order for `columns`. Always `'asc'`. */ + readonly scanDirection: 'asc'; +} + +/** One `viewer_anchor_facts` row's worth of keyset column values. */ +export interface InboundLinksKeysetRow { + /** See `viewer_anchor_facts.edge_id`. */ + edge_id: number; +} + +/** + * The subset of `ListInboundLinksOptions` that affects which rows match — + * used to build a cursor's `filterKey` so a cursor minted for one page's + * inbound links can't silently be replayed against another's. + */ +export interface InboundLinksCursorFilterKeyInput { + /** The resolved canonical `content_items.id` whose inbound links these are. */ + destPageId: number; +} + +/** + * Decoded shape of an opaque `/api/pages/inbound-links` viewer cursor. + */ +export interface InboundLinksCursorPayload { + /** + * The read-model schema version the cursor was minted under (see + * `VIEWER_READ_MODEL_SCHEMA_VERSION`). + */ + v: number; + /** See `buildInboundLinksFilterKey`. */ + filterKey: string; + /** The sort field the cursor was minted under — always `'edgeId'`. */ + sortBy: 'edgeId'; + /** + * The sort direction the cursor was minted under — always `'asc'` at + * runtime (see {@link InboundLinksSortSpec}'s docs), but typed as the + * shared `'asc' | 'desc'` union to match `decodeCursorEnvelope`'s + * generic `CursorEnvelope` return shape. + */ + sortOrder: 'asc' | 'desc'; + /** The boundary row's keyset tuple values, in sort-spec column order. */ + values: (string | number)[]; +} diff --git a/packages/@nitpicker/query/src/viewer-read-model/compute-anchor-fact-rows.spec.ts b/packages/@nitpicker/query/src/viewer-read-model/compute-anchor-fact-rows.spec.ts index c342b21c..d858b47f 100644 --- a/packages/@nitpicker/query/src/viewer-read-model/compute-anchor-fact-rows.spec.ts +++ b/packages/@nitpicker/query/src/viewer-read-model/compute-anchor-fact-rows.spec.ts @@ -318,6 +318,185 @@ describe('computeAnchorFactRows', () => { }); }); +/** + * `first_text_id` is `MIN(ae.first_text_id)` over every `anchor_edges` row + * collapsed into one `viewer_anchor_facts` row. A single referrer with two + * distinct anchors — one directly at the canonical destination, one at a + * redirect source that lands on it — produces two distinct `anchor_edges` + * rows (different `href_page_id`) that the redirect-resolution `GROUP BY` + * collapses into one, exercising the `MIN()` aggregation across rows rather + * than the trivial single-row passthrough. + */ +describe('computeAnchorFactRows — first_text_id', () => { + const workingDir = path.resolve( + __dirname, + '__test_fixtures_compute_anchor_fact_rows_first_text_id__', + ); + let archive: InstanceType; + const archiveFilePath = path.resolve( + workingDir, + 'compute-anchor-fact-rows-first-text-id-test.nitpicker', + ); + + beforeAll(async () => { + const { mkdirSync } = await import('node:fs'); + mkdirSync(workingDir, { recursive: true }); + archive = await Archive.create({ filePath: archiveFilePath, cwd: workingDir }); + await archive.setConfig(BASE_CONFIG); + + // One referrer with a single anchor carrying no text — the anchor's + // `textContent` is `null`, so `anchor_edges.first_text_id` is `null` + // for this (page, href) pair (crawler layer, tested separately). + await archive.setPage({ + url: parseUrl('https://example.com/textless-referrer')!, + redirectPaths: [], + isExternal: false, + isTarget: true, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 100, + responseHeaders: {}, + html: '', + meta: { ...META, title: 'Textless referrer' }, + anchorList: [ + { + href: parseUrl('https://example.com/textless-target')!, + isExternal: false, + title: null, + textContent: null, + }, + ], + imageList: [], + isSkipped: false, + }); + await archive.setPage({ + url: parseUrl('https://example.com/textless-target')!, + redirectPaths: [], + isExternal: false, + isTarget: true, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 100, + responseHeaders: {}, + html: '', + meta: META, + anchorList: [], + imageList: [], + isSkipped: false, + }); + + // One referrer linking the same canonical destination twice: once + // directly, once via a redirect source — two distinct `anchor_edges` + // rows that collapse into one `viewer_anchor_facts` row, so + // `first_text_id` must resolve to the earlier-inserted anchor's text + // ("Direct link", inserted first in `anchorList`), not the later one. + await archive.setPage({ + url: parseUrl('https://example.com/multi-anchor-referrer')!, + redirectPaths: [], + isExternal: false, + isTarget: true, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 100, + responseHeaders: {}, + html: '', + meta: { ...META, title: 'Multi-anchor referrer' }, + anchorList: [ + { + href: parseUrl('https://example.com/canonical')!, + isExternal: false, + title: null, + textContent: 'Direct link', + }, + { + href: parseUrl('https://example.com/old-path')!, + isExternal: false, + title: null, + textContent: 'Via redirect source', + }, + ], + imageList: [], + isSkipped: false, + }); + await archive.setPage({ + url: parseUrl('https://example.com/canonical')!, + redirectPaths: [], + isExternal: false, + isTarget: true, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 100, + responseHeaders: {}, + html: '', + meta: { ...META, title: 'Canonical' }, + anchorList: [], + imageList: [], + isSkipped: false, + }); + await archive.setRedirect({ + url: parseUrl('https://example.com/old-path')!, + redirectPaths: ['https://example.com/canonical'], + isExternal: false, + isTarget: true, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 100, + responseHeaders: {}, + html: '', + meta: META, + anchorList: [], + imageList: [], + isSkipped: false, + }); + }); + + afterAll(async () => { + if (archive) { + await archive.releaseHandle(); + } + const { rmSync } = await import('node:fs'); + rmSync(workingDir, { recursive: true, force: true }); + }); + + it('sets first_text_id to null when the anchor carried no text', async () => { + const knex = archive.getKnex(); + const { rows, urlByRefId } = await knex.transaction((trx) => + collectAnchorFactRows(trx), + ); + const textless = findByDestUrl( + rows, + urlByRefId, + 'https://example.com/textless-target', + )!; + expect(textless.first_text_id).toBeNull(); + }); + + it('resolves first_text_id, via text_refs, to the anchor text of the target', async () => { + const knex = archive.getKnex(); + const { rows, urlByRefId } = await knex.transaction((trx) => + collectAnchorFactRows(trx), + ); + const direct = findByDestUrl(rows, urlByRefId, 'https://example.com/canonical')!; + expect(direct.first_text_id).not.toBeNull(); + const textRow = await knex('text_refs').where('id', direct.first_text_id).first(); + expect(textRow?.text).toBe('Direct link'); + }); + + it('collapses the direct anchor and the redirect-source anchor from the same referrer into one row, count: 2', async () => { + const knex = archive.getKnex(); + const { rows, urlByRefId } = await knex.transaction((trx) => + collectAnchorFactRows(trx), + ); + const direct = findByDestUrl(rows, urlByRefId, 'https://example.com/canonical')!; + expect(direct.count).toBe(2); + }); +}); + /** * Mirrors `list-links.spec.ts`'s redirect-resolution describe block: an * anchor to an internal redirect-source page and an anchor directly to the diff --git a/packages/@nitpicker/query/src/viewer-read-model/compute-anchor-fact-rows.ts b/packages/@nitpicker/query/src/viewer-read-model/compute-anchor-fact-rows.ts index 614f2ddb..ee3e2461 100644 --- a/packages/@nitpicker/query/src/viewer-read-model/compute-anchor-fact-rows.ts +++ b/packages/@nitpicker/query/src/viewer-read-model/compute-anchor-fact-rows.ts @@ -26,7 +26,19 @@ const READ_CHUNK_SIZE = 2000; * resolved-canonical destinations: multiple distinct dest pages that * redirect to the same canonical collapse into one output row and their * counts must add — the same summed occurrence count a `count(*)` over - * per-row anchor rows yields. + * per-row anchor rows yields. `first_text_id` is `MIN(ae.first_text_id)` + * over the same group, so `listInboundLinks` can read anchor text straight + * off `viewer_anchor_facts` without a second `anchor_edges` round-trip. When + * the group is a single `anchor_edges` row (the common case — one referrer, + * one href), this exactly mirrors that row's own first-wins anchor text. + * When redirect/alias resolution collapses *distinct* `anchor_edges` rows + * from the same referrer into one group (e.g. one anchor points directly at + * the canonical destination and another points at a redirect source that + * lands on it), `MIN()` picks the lower `text_refs.id` — a deterministic, + * stable choice across rebuilds, but not a guarantee that it's the + * chronologically-first-observed anchor among the distinct hrefs (unlike the + * single-row case, `text_refs.id` order need not track anchor-array order + * once ids are shared/reused elsewhere in the archive). * * Reads `anchor_edges` in bounded chunks by partitioning `source.id` * (`content_items.id`) into non-overlapping ranges @@ -105,6 +117,7 @@ export async function* computeAnchorFactRows( status: number | null; isExternal: 0 | 1; count: number; + firstTextId: number | null; }[] = await trx('anchor_edges as ae') .join('content_items as source', 'ae.page_id', '=', 'source.id') .join('content_items as dest', 'ae.href_page_id', '=', 'dest.id') @@ -156,6 +169,7 @@ export async function* computeAnchorFactRows( trx.raw(`${statusExpression} as "status"`), trx.raw(`${isExternalExpression} as "isExternal"`), trx.raw('sum("ae"."count") as "count"'), + trx.raw('min("ae"."first_text_id") as "firstTextId"'), ); if (rows.length === 0) { @@ -180,6 +194,7 @@ export async function* computeAnchorFactRows( count: Number(row.count), is_broken: row.status === 404 ? 1 : 0, is_external_link: row.isExternal ? 1 : 0, + first_text_id: row.firstTextId, }; }); } diff --git a/packages/@nitpicker/query/src/viewer-read-model/create-viewer-read-model-tables.ts b/packages/@nitpicker/query/src/viewer-read-model/create-viewer-read-model-tables.ts index 6d27ee67..62f091ae 100644 --- a/packages/@nitpicker/query/src/viewer-read-model/create-viewer-read-model-tables.ts +++ b/packages/@nitpicker/query/src/viewer-read-model/create-viewer-read-model-tables.ts @@ -216,6 +216,12 @@ export async function createViewerReadModelTables(trx: Knex): Promise { // primary column is negated and walked ascending instead. See // ARCHITECTURE.md「設計注意（viewer_anchor_facts read model、issue // #114）」for the full read/write/storage rationale. + // `first_text_id` mirrors `anchor_edges.first_text_id` (first-wins anchor + // text for the pair, see the anchor_edges dedup invariant) — references + // `text_refs(id)` directly rather than `viewer_url_refs`/a dedicated + // viewer-side copy, since text_refs already exists in the same database + // and duplicating its rows here would cost storage for no read-path + // benefit (this table never seeks by text). await trx.raw(` CREATE TABLE viewer_anchor_facts ( edge_id integer primary key, @@ -228,7 +234,8 @@ export async function createViewerReadModelTables(trx: Knex): Promise { status_desc_key integer not null, count integer not null, is_broken integer not null, - is_external_link integer not null + is_external_link integer not null, + first_text_id integer references text_refs(id) ) `); diff --git a/packages/@nitpicker/query/src/viewer-read-model/types.ts b/packages/@nitpicker/query/src/viewer-read-model/types.ts index 58a08633..8aac6c5d 100644 --- a/packages/@nitpicker/query/src/viewer-read-model/types.ts +++ b/packages/@nitpicker/query/src/viewer-read-model/types.ts @@ -195,7 +195,7 @@ export interface ExternalLinkInsertRow { * the count of {@link AnchorFactInsertRow} rows sharing this * `dest_page_id`, since those rows are already deduplicated one-per- * `(source_page_id, dest_page_id)` pair. Must stay in the same counting - * grain as `getPageDetail.inboundLinks` (see that function's docs, #71) + * grain as `listInboundLinks`'s `total` (see that function's docs, #235) * — multiple anchors from the same page count once. */ referrer_count: number; @@ -238,6 +238,14 @@ export interface AnchorFactInsertRow { * time, never by an indexed read query. */ is_external_link: number; + /** + * `MIN(ae.first_text_id)` — the first-wins anchor text's `text_refs.id` + * for this `(source_page_id, dest_page_id)` pair (same first-wins + * semantics as `anchor_edges.first_text_id`), or `null` if the anchor + * carried no text. Read directly by `listInboundLinks` so a referrer + * window's anchor text never needs a second `anchor_edges` round-trip. + */ + first_text_id: number | null; } /** diff --git a/packages/@nitpicker/query/src/viewer-read-model/viewer-read-model-schema-version.ts b/packages/@nitpicker/query/src/viewer-read-model/viewer-read-model-schema-version.ts index 589db4dd..d93f5989 100644 --- a/packages/@nitpicker/query/src/viewer-read-model/viewer-read-model-schema-version.ts +++ b/packages/@nitpicker/query/src/viewer-read-model/viewer-read-model-schema-version.ts @@ -7,4 +7,4 @@ * `viewer_read_model_meta.schema_version` to decide whether a rebuild is * needed. */ -export const VIEWER_READ_MODEL_SCHEMA_VERSION = 21; +export const VIEWER_READ_MODEL_SCHEMA_VERSION = 22; diff --git a/packages/@nitpicker/viewer/e2e/generate-inbound-links-fixture.mjs b/packages/@nitpicker/viewer/e2e/generate-inbound-links-fixture.mjs new file mode 100644 index 00000000..63eb975c --- /dev/null +++ b/packages/@nitpicker/viewer/e2e/generate-inbound-links-fixture.mjs @@ -0,0 +1,135 @@ +import { mkdirSync, rmSync } from 'node:fs'; +import path from 'node:path'; + +import { tryParseUrl as parseUrl } from '@d-zero/shared/parse-url'; +import { Archive } from '@nitpicker/crawler'; +import { buildViewerReadModel } from '@nitpicker/query'; + +const dirname = import.meta.dirname; +const FIXTURE_PATH = path.resolve(dirname, '.fixture-inbound-links.nitpicker'); +const FIXTURE_CWD = path.resolve(dirname, '.fixture-inbound-links-tmp'); + +rmSync(FIXTURE_PATH, { force: true }); +rmSync(FIXTURE_CWD, { recursive: true, force: true }); +mkdirSync(FIXTURE_CWD, { recursive: true }); + +const archive = await Archive.create({ filePath: FIXTURE_PATH, cwd: FIXTURE_CWD }); +await archive.setConfig({ + baseUrl: 'https://example.com', + name: 'e2e-inbound-links-fixture', + version: '0.13.0', + recursive: true, + interval: 0, + image: true, + fetchExternal: false, + parallels: 1, + roots: ['https://example.com'], + excludes: [], + excludeKeywords: [], + excludeUrls: [], + maxExcludedDepth: 0, + retry: 3, + fromList: false, + disableQueries: false, + userAgent: 'e2e', + ignoreRobots: false, +}); + +const NO_META = { + lang: 'ja', + title: null, + description: null, + keywords: null, + noindex: false, + nofollow: false, + noarchive: false, + canonical: null, + alternate: null, + 'og:type': null, + 'og:title': null, + 'og:site_name': null, + 'og:description': null, + 'og:url': null, + 'og:image': null, + 'twitter:card': null, +}; +// Must exceed the default MPA `pageSize=100` so the inbound-links list +// paginates (Next button enables, second page has real rows) — mirrors +// `generate-fixture.mjs`'s own `PAGE_COUNT` rationale. +const REFERRER_COUNT = 101; + +await archive.setPage({ + url: parseUrl('https://example.com/target'), + redirectPaths: [], + isExternal: false, + isTarget: true, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 200, + responseHeaders: {}, + html: 'Target

Target

', + meta: { ...NO_META, title: 'Target' }, + anchorList: [], + imageList: [], + isSkipped: false, +}); + +// A second target with zero referrers, for the "no inbound links" empty state. +await archive.setPage({ + url: parseUrl('https://example.com/lonely-target'), + redirectPaths: [], + isExternal: false, + isTarget: true, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 200, + responseHeaders: {}, + html: 'Lonely Target

Lonely Target

', + meta: { ...NO_META, title: 'Lonely Target' }, + anchorList: [], + imageList: [], + isSkipped: false, +}); + +for (let i = 0; i < REFERRER_COUNT; i++) { + const name = `referrer-${i}`; + await archive.setPage({ + url: parseUrl(`https://example.com/${name}`), + redirectPaths: [], + isExternal: false, + isTarget: true, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 200, + responseHeaders: {}, + html: `${name}

${name}

`, + meta: { ...NO_META, title: name }, + anchorList: [ + { + href: parseUrl('https://example.com/target'), + isExternal: false, + title: null, + textContent: `Link from ${name}`, + }, + ], + imageList: [], + isSkipped: false, + }); +} + +// Build the viewer read model before writing — `listInboundLinks` (unlike +// most viewer_*-backed queries) has no legacy fallback, so `/api/pages/inbound-links` +// would otherwise always respond `{ available: false }` against this +// fixture. Kept in its own fixture (rather than adding this to the shared +// `generate-fixture.mjs`) for the same reason `generate-directory-tree-fixture.mjs` +// is separate: the shared fixture's directory-tree "no read model" empty-state +// test needs the read model to stay unbuilt. +await buildViewerReadModel(archive); + +await archive.write(); +await archive.close(); +// eslint-disable-next-line no-console +console.log(`E2E inbound-links fixture created: ${FIXTURE_PATH}`); diff --git a/packages/@nitpicker/viewer/e2e/inbound-links.spec.ts b/packages/@nitpicker/viewer/e2e/inbound-links.spec.ts new file mode 100644 index 00000000..bd753a2e --- /dev/null +++ b/packages/@nitpicker/viewer/e2e/inbound-links.spec.ts @@ -0,0 +1,90 @@ +import { expect, test } from '@playwright/test'; + +/** + * Covers issue #235's split of inbound links out of Page Detail into a + * dedicated, paginated view, against the dedicated fixture (see + * `generate-inbound-links-fixture.mjs` — its `viewer_anchor_facts` read + * model is built explicitly, unlike the shared `generate-fixture.mjs` + * fixture used by `viewer.spec.ts`). + * + * Fixture layout: `/target` has 101 referrers (`/referrer-0`..`/referrer-100`, + * exceeding the default MPA page size so Next/virtual-scroll pagination has + * a real second page), and `/lonely-target` has none. + */ +test.describe('Nitpicker Viewer inbound links', () => { + test('Page Detail は被リンク件数とリンクを表示し、クリックで一覧ビューに遷移する', async ({ + page, + }) => { + await page.goto( + `/pages/detail?url=${encodeURIComponent('https://example.com/target')}`, + ); + await expect( + page.getByRole('heading', { name: 'Page detail', level: 1 }), + ).toBeVisible(); + await expect(page.getByText('Inbound links (101)')).toBeVisible(); + + await page.getByRole('link', { name: 'View all inbound links' }).click(); + + await expect( + page.getByRole('heading', { name: 'Inbound links', level: 1 }), + ).toBeVisible(); + await expect(page.getByText('https://example.com/referrer-0')).toBeVisible(); + }); + + test('MPA ページネーションで Next を押すと2ページ目の行が読み込まれる', async ({ + page, + }) => { + await page.goto( + `/pages/inbound-links?url=${encodeURIComponent('https://example.com/target')}`, + ); + await expect(page.locator('.pt-row').first()).toBeVisible(); + await expect(page.locator('.pt-row')).toHaveCount(100); + const next = page.getByRole('button', { name: 'Next' }); + await next.click(); + await expect(page).toHaveURL(/[?&]page=2(?:&|$)/); + await expect(page.locator('.pt-row').first()).toBeVisible(); + await expect(page.locator('.pt-row')).toHaveCount(1); + }); + + test('Page Detail から被リンク0件のページでは一覧へのリンクが表示されない', async ({ + page, + }) => { + await page.goto( + `/pages/detail?url=${encodeURIComponent('https://example.com/lonely-target')}`, + ); + await expect(page.getByRole('heading', { name: 'Inbound links (0)' })).toBeVisible(); + await expect(page.getByRole('link', { name: 'View all inbound links' })).toHaveCount( + 0, + ); + }); + + test('存在しない URL への直接アクセスではページが見つからないエラーが表示される', async ({ + page, + }) => { + await page.goto( + `/pages/inbound-links?url=${encodeURIComponent('https://example.com/nonexistent')}`, + ); + await expect( + page.getByRole('heading', { name: 'Inbound links', level: 1 }), + ).toBeVisible(); + await expect(page.getByText(/Page not found/i)).toBeVisible(); + }); + + test.describe('virtual scroll', () => { + test.beforeEach(async ({ page }) => { + // Pin the localStorage preference *before* the SPA loads so the + // first render is already in virtual mode. + await page.addInitScript(() => { + globalThis.localStorage.setItem('nitpicker-pagination-mode', 'virtual'); + }); + }); + + test('被リンク一覧が仮想スクロールで表示される', async ({ page }) => { + await page.goto( + `/pages/inbound-links?url=${encodeURIComponent('https://example.com/target')}`, + ); + await expect(page.locator('.vt-row').first()).toBeVisible(); + await expect(page.locator('.vt-meta')).toContainText('101'); + }); + }); +}); diff --git a/packages/@nitpicker/viewer/e2e/viewer-stub.spec.ts b/packages/@nitpicker/viewer/e2e/viewer-stub.spec.ts index bc80b43e..711fecba 100644 --- a/packages/@nitpicker/viewer/e2e/viewer-stub.spec.ts +++ b/packages/@nitpicker/viewer/e2e/viewer-stub.spec.ts @@ -55,6 +55,12 @@ test.describe('Nitpicker Viewer (stub mode)', () => { await expect( page.getByRole('heading', { name: 'Page detail', level: 1 }), ).toBeVisible(); + // `viewer_anchor_facts` can never exist in stub mode (`buildViewerReadModel` + // refuses read-only accessors, and `viewer-build` refuses stub + // directories) — `/api/pages/inbound-links` must respond with the + // `{ available: false }` marker rather than the route throwing, so + // Page Detail itself never 500s here (issue #235). + await expect(page.getByText('Available once the crawl finishes.')).toBeVisible(); }); test('viewer 起動中も stub の tmpDir は残存し、.nitpicker は未生成', () => { diff --git a/packages/@nitpicker/viewer/e2e/viewer.spec.ts b/packages/@nitpicker/viewer/e2e/viewer.spec.ts index 120d1e73..0de82865 100644 --- a/packages/@nitpicker/viewer/e2e/viewer.spec.ts +++ b/packages/@nitpicker/viewer/e2e/viewer.spec.ts @@ -106,7 +106,7 @@ test.describe('Nitpicker Viewer', () => { await expect(page.locator('.pt-row').first()).toContainText('2'); }); - test('外部リンクの宛先をクリックすると Page Detail で参照元ページ一覧が確認できる', async ({ + test('外部リンクの宛先をクリックすると Page Detail から被リンクへの導線が確認できる', async ({ page, }) => { await page.goto('/external-links'); @@ -114,10 +114,17 @@ test.describe('Nitpicker Viewer', () => { await expect( page.getByRole('heading', { name: 'Page detail', level: 1 }), ).toBeVisible(); - // Two internal pages link to this external destination. - await expect( - page.getByRole('heading', { name: /Inbound links \(2\)/ }), - ).toBeVisible(); + // This suite's shared fixture (generate-fixture.mjs) never calls + // buildViewerReadModel (see its own docs — directory-tree's "no read + // model" empty-state test depends on that), and `listInboundLinks` + // has no legacy fallback, so the inbound-links count surfaces the + // actionable "run viewer-build" error here instead of a live count. + // The live count (2 internal pages link to this destination) and the + // full inbound-links list have their own coverage in + // `inbound-links.spec.ts`, against a fixture that builds the read + // model. + await expect(page.getByRole('heading', { name: 'Inbound links' })).toBeVisible(); + await expect(page.getByText(/viewer-build/)).toBeVisible(); // External pages are never scraped — the HTML snapshot / outbound // links sections are not meaningful and must not render. await expect(page.getByRole('heading', { name: /Outbound links/ })).toHaveCount(0); diff --git a/packages/@nitpicker/viewer/package.json b/packages/@nitpicker/viewer/package.json index da7ce2fe..385e39c4 100644 --- a/packages/@nitpicker/viewer/package.json +++ b/packages/@nitpicker/viewer/package.json @@ -32,7 +32,8 @@ "test:e2e": "node ./e2e/generate-fixture.mjs && playwright test", "test:e2e:stub": "node ./e2e/generate-stub-fixture.mjs && playwright test --config playwright.stub.config.ts", "test:e2e:directory-tree": "node ./e2e/generate-directory-tree-fixture.mjs && playwright test --config playwright.directory-tree.config.ts", - "test:e2e:template-clusters": "node ./e2e/generate-template-clusters-fixture.mjs && playwright test --config playwright.template-clusters.config.ts" + "test:e2e:template-clusters": "node ./e2e/generate-template-clusters-fixture.mjs && playwright test --config playwright.template-clusters.config.ts", + "test:e2e:inbound-links": "node ./e2e/generate-inbound-links-fixture.mjs && playwright test --config playwright.inbound-links.config.ts" }, "dependencies": { "@d-zero/dealer": "1.9.4", diff --git a/packages/@nitpicker/viewer/playwright.config.ts b/packages/@nitpicker/viewer/playwright.config.ts index c479eed3..2cfe2a99 100644 --- a/packages/@nitpicker/viewer/playwright.config.ts +++ b/packages/@nitpicker/viewer/playwright.config.ts @@ -19,18 +19,20 @@ const PORT = 4325; */ export default defineConfig({ testDir: './e2e', - // The stub-mode, directory-tree, and template-clusters-classified suites - // each have their own webServer (a different fixture and port) and are - // wired up via `playwright.stub.config.ts` / `test:e2e:stub`, - // `playwright.directory-tree.config.ts` / `test:e2e:directory-tree`, and - // `playwright.template-clusters.config.ts` / `test:e2e:template-clusters`. - // Keep them out of this run so every dedicated-fixture suite stays - // independently scheduled in CI — without this, e.g. - // `directory-tree.spec.ts` would also run here against the shared - // fixture, whose `/api/directory-tree` always returns an empty + // The stub-mode, directory-tree, template-clusters-classified, and + // inbound-links suites each have their own webServer (a different fixture + // and port) and are wired up via `playwright.stub.config.ts` / + // `test:e2e:stub`, `playwright.directory-tree.config.ts` / + // `test:e2e:directory-tree`, `playwright.template-clusters.config.ts` / + // `test:e2e:template-clusters`, and `playwright.inbound-links.config.ts` / + // `test:e2e:inbound-links`. Keep them out of this run so every + // dedicated-fixture suite stays independently scheduled in CI — without + // this, e.g. `directory-tree.spec.ts` would also run here against the + // shared fixture, whose `/api/directory-tree` always returns an empty // `{ roots: [] }` (the shared fixture never builds the viewer read // model), and every assertion would fail. - testIgnore: /(viewer-stub|directory-tree|template-clusters-classified)\.spec\.ts$/, + testIgnore: + /(viewer-stub|directory-tree|template-clusters-classified|inbound-links)\.spec\.ts$/, fullyParallel: false, workers: 1, retries: 0, diff --git a/packages/@nitpicker/viewer/playwright.inbound-links.config.ts b/packages/@nitpicker/viewer/playwright.inbound-links.config.ts new file mode 100644 index 00000000..37617a65 --- /dev/null +++ b/packages/@nitpicker/viewer/playwright.inbound-links.config.ts @@ -0,0 +1,52 @@ +import { existsSync } from 'node:fs'; +import path from 'node:path'; + +import { defineConfig, devices } from '@playwright/test'; + +const dirname = import.meta.dirname; +const fixturePath = path.resolve(dirname, 'e2e/.fixture-inbound-links.nitpicker'); +const cliBin = path.resolve(dirname, '../cli/bin/nitpicker.js'); + +/** Port the inbound-links-mode viewer server listens on during E2E. */ +const PORT = 4329; + +/** + * Playwright configuration for the **inbound links** Viewer E2E suite. + * + * Sibling of `playwright.config.ts` / `playwright.stub.config.ts` / + * `playwright.directory-tree.config.ts` / `playwright.template-clusters.config.ts`: + * same SPA, same CLI bin, but the `webServer` points at a fixture built with + * `e2e/generate-inbound-links-fixture.mjs`, whose viewer read model is built + * before writing — `listInboundLinks` has no legacy fallback (unlike most + * `viewer_*`-backed queries), so `/api/pages/inbound-links` would otherwise + * always respond `{ available: false }` against the shared + * `generate-fixture.mjs` fixture, which intentionally never builds the read + * model (its own directory-tree "no read model" empty-state test depends on + * that). + */ +if (!existsSync(fixturePath)) { + throw new Error( + `Inbound-links fixture not found: ${fixturePath}. Run e2e/generate-inbound-links-fixture.mjs first.`, + ); +} + +export default defineConfig({ + testDir: './e2e', + testMatch: /inbound-links\.spec\.ts$/, + fullyParallel: false, + workers: 1, + retries: 0, + reporter: 'list', + webServer: { + command: `node ${cliBin} viewer ${fixturePath} --no-open --port ${PORT}`, + url: `http://localhost:${PORT}`, + reuseExistingServer: false, + timeout: 60_000, + }, + use: { + baseURL: `http://localhost:${PORT}`, + locale: 'en-US', + trace: 'on-first-retry', + }, + projects: [{ name: 'chromium', use: { ...devices['Desktop Chrome'] } }], +}); diff --git a/packages/@nitpicker/viewer/src/create-app.ts b/packages/@nitpicker/viewer/src/create-app.ts index de20a7ab..6448fbc0 100644 --- a/packages/@nitpicker/viewer/src/create-app.ts +++ b/packages/@nitpicker/viewer/src/create-app.ts @@ -13,6 +13,7 @@ import { registerErrorKindsRoute } from './routes/register-error-kinds-route.js' import { registerGraphRoute } from './routes/register-graph-route.js'; import { registerHeaderChecksRoute } from './routes/register-header-checks-route.js'; import { registerImagesRoute } from './routes/register-images-route.js'; +import { registerInboundLinksRoute } from './routes/register-inbound-links-route.js'; import { registerIsolatedClustersRoute } from './routes/register-isolated-clusters-route.js'; import { registerIsolatedPagesRoute } from './routes/register-isolated-pages-route.js'; import { registerLinksRoute } from './routes/register-links-route.js'; @@ -52,6 +53,7 @@ export function createApp(options: CreateAppOptions): Hono { registerSummaryRoute(app, context); registerPagesRoute(app, context); registerPageDetailRoute(app, context); + registerInboundLinksRoute(app, context); registerPageHtmlRoute(app, context); registerPageMainContentsRoute(app, context); registerDirectoryTreeRoute(app, context); diff --git a/packages/@nitpicker/viewer/src/routes/register-inbound-links-route.spec.ts b/packages/@nitpicker/viewer/src/routes/register-inbound-links-route.spec.ts new file mode 100644 index 00000000..62da3741 --- /dev/null +++ b/packages/@nitpicker/viewer/src/routes/register-inbound-links-route.spec.ts @@ -0,0 +1,224 @@ +import type { ArchiveContext } from '../types.js'; +import type { ArchiveManager } from '@nitpicker/query'; + +import path from 'node:path'; + +import { tryParseUrl as parseUrl } from '@d-zero/shared/parse-url'; +import { Archive } from '@nitpicker/crawler'; +import { buildViewerReadModel } from '@nitpicker/query'; +import { afterAll, beforeAll, describe, expect, it } from 'vitest'; + +import { createApp } from '../create-app.js'; + +const __filename = new URL(import.meta.url).pathname; +const __dirname = path.dirname(__filename); +const workingDir = path.resolve( + __dirname, + '__test_fixtures_register_inbound_links_route__', +); + +const BASE_CONFIG = { + baseUrl: 'https://example.com', + name: 'test', + version: '0.13.0', + recursive: true, + interval: 0, + image: true, + fetchExternal: false, + parallels: 1, + roots: ['https://example.com'], + excludes: [], + excludeKeywords: [], + excludeUrls: [], + maxExcludedDepth: 0, + retry: 3, + fromList: false, + disableQueries: false, + userAgent: 'test', + ignoreRobots: false, +}; + +const META = { + lang: null, + title: null, + description: null, + keywords: null, + noindex: false, + nofollow: false, + noarchive: false, + canonical: null, + alternate: null, + 'og:type': null, + 'og:title': null, + 'og:site_name': null, + 'og:description': null, + 'og:url': null, + 'og:image': null, + 'twitter:card': null, +}; + +describe('registerInboundLinksRoute — /api/pages/inbound-links (integration)', () => { + describe('archive mode (viewer_anchor_facts read model built)', () => { + let archive: InstanceType; + let app: ReturnType; + const archiveFilePath = path.resolve(workingDir, 'fixture.nitpicker'); + + beforeAll(async () => { + const { mkdirSync } = await import('node:fs'); + mkdirSync(workingDir, { recursive: true }); + archive = await Archive.create({ filePath: archiveFilePath, cwd: workingDir }); + await archive.setConfig(BASE_CONFIG); + + await archive.setPage({ + url: parseUrl('https://example.com/target')!, + redirectPaths: [], + isExternal: false, + isTarget: true, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 100, + responseHeaders: {}, + html: '', + meta: META, + anchorList: [], + imageList: [], + isSkipped: false, + }); + for (const name of ['referrer-a', 'referrer-b']) { + await archive.setPage({ + url: parseUrl(`https://example.com/${name}`)!, + redirectPaths: [], + isExternal: false, + isTarget: true, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 100, + responseHeaders: {}, + html: '', + meta: META, + anchorList: [ + { + href: parseUrl('https://example.com/target')!, + isExternal: false, + title: null, + textContent: `Link from ${name}`, + }, + ], + imageList: [], + isSkipped: false, + }); + } + + await buildViewerReadModel(archive); + + const context: ArchiveContext = { + manager: { get: () => archive } as unknown as ArchiveManager, + archiveId: 'test', + filePath: archiveFilePath, + mode: 'archive', + crawlerLockHolder: null, + }; + app = createApp({ + context, + publicDir: '/tmp/no-such-dir-register-inbound-links-route-spec', + }); + }); + + afterAll(async () => { + await archive.close(); + const { rmSync } = await import('node:fs'); + rmSync(workingDir, { recursive: true, force: true }); + }); + + it('requires the url query parameter', async () => { + const res = await app.request('/api/pages/inbound-links'); + expect(res.status).toBe(400); + }); + + it('returns 404 for a URL that does not exist', async () => { + const res = await app.request( + '/api/pages/inbound-links?url=https://example.com/missing', + ); + expect(res.status).toBe(404); + }); + + it('returns every referrer with anchor text and count within the default limit', async () => { + const res = await app.request( + '/api/pages/inbound-links?url=https://example.com/target', + ); + const body = (await res.json()) as { + url: string; + items: { url: string; textContent: string | null; count: number }[]; + total: number; + nextCursor: string | null; + }; + expect(body.total).toBe(2); + expect(body.items).toHaveLength(2); + expect(body.items.toSorted((a, b) => a.url.localeCompare(b.url))).toEqual([ + { + url: 'https://example.com/referrer-a', + textContent: 'Link from referrer-a', + count: 1, + }, + { + url: 'https://example.com/referrer-b', + textContent: 'Link from referrer-b', + count: 1, + }, + ]); + expect(body.nextCursor).toBeNull(); + }); + + it('bounds by the limit query parameter and continues via nextCursor', async () => { + const first = await app.request( + '/api/pages/inbound-links?url=https://example.com/target&limit=1', + ); + const firstBody = (await first.json()) as { + items: unknown[]; + nextCursor: string | null; + }; + expect(firstBody.items).toHaveLength(1); + expect(firstBody.nextCursor).not.toBeNull(); + + const second = await app.request( + `/api/pages/inbound-links?url=https://example.com/target&limit=1&cursor=${firstBody.nextCursor}`, + ); + const secondBody = (await second.json()) as { + items: unknown[]; + nextCursor: string | null; + }; + expect(secondBody.items).toHaveLength(1); + expect(secondBody.nextCursor).toBeNull(); + }); + + it('returns only the total when limit=0, skipping the row window', async () => { + const res = await app.request( + '/api/pages/inbound-links?url=https://example.com/target&limit=0', + ); + const body = (await res.json()) as { items: unknown[]; total: number }; + expect(body.total).toBe(2); + expect(body.items).toHaveLength(0); + }); + }); + + describe('stub mode (live crawl — viewer_anchor_facts can never exist)', () => { + it('responds with available: false instead of attempting a query that would throw', async () => { + const fakeArchive = {} as unknown as Archive; + const context: ArchiveContext = { + manager: { get: () => fakeArchive } as unknown as ArchiveManager, + archiveId: 'live-stub', + filePath: '/tmp/._nitpicker-live', + mode: 'stub', + crawlerLockHolder: null, + }; + const app = createApp({ context, publicDir: workingDir }); + const res = await app.request( + '/api/pages/inbound-links?url=https://example.com/target', + ); + expect(res.status).toBe(200); + expect(await res.json()).toEqual({ available: false }); + }); + }); +}); diff --git a/packages/@nitpicker/viewer/src/routes/register-inbound-links-route.ts b/packages/@nitpicker/viewer/src/routes/register-inbound-links-route.ts new file mode 100644 index 00000000..554398ab --- /dev/null +++ b/packages/@nitpicker/viewer/src/routes/register-inbound-links-route.ts @@ -0,0 +1,45 @@ +import type { ArchiveContext, InboundLinksUnavailable } from '../types.js'; +import type { InboundLinkList } from '@nitpicker/query'; +import type { Hono } from 'hono'; + +import { listInboundLinks } from '@nitpicker/query'; + +import { toNumber } from '../query-params/to-number.js'; + +/** + * Registers `GET /api/pages/inbound-links?url=&limit=&offset=&cursor=&direction=` — + * a bounded, cursor-paginated window of pages linking to a target page, plus + * its total referrer count. + * + * There is no legacy fallback (unlike `/api/links` and most other + * `viewer_*`-backed routes): `listInboundLinks` reads exclusively from + * `viewer_anchor_facts`, so in stub mode — where that read model cannot + * exist — this responds with {@link InboundLinksUnavailable} instead of + * attempting a query that would only throw. + * @param app - The Hono application. + * @param context - The opened archive context. + */ +export function registerInboundLinksRoute(app: Hono, context: ArchiveContext): void { + app.get('/api/pages/inbound-links', async (c) => { + const url = c.req.query('url'); + if (!url) { + return c.json({ error: 'Missing required query parameter: url' }, 400); + } + if (context.mode === 'stub') { + const unavailable: InboundLinksUnavailable = { available: false }; + return c.json(unavailable); + } + const accessor = context.manager.get(context.archiveId); + const result: InboundLinkList | null = await listInboundLinks(accessor, { + url, + limit: toNumber(c.req.query('limit')), + offset: toNumber(c.req.query('offset')), + cursor: c.req.query('cursor'), + direction: c.req.query('direction') === 'prev' ? 'prev' : undefined, + }); + if (!result) { + return c.json({ error: 'Page not found' }, 404); + } + return c.json(result); + }); +} diff --git a/packages/@nitpicker/viewer/src/types.ts b/packages/@nitpicker/viewer/src/types.ts index c2081538..14ec3b85 100644 --- a/packages/@nitpicker/viewer/src/types.ts +++ b/packages/@nitpicker/viewer/src/types.ts @@ -49,3 +49,18 @@ export interface CreateAppOptions { /** Absolute path to the directory containing the built frontend assets. */ publicDir: string; } + +/** + * Response shape for `/api/pages/inbound-links` when the viewer read model + * cannot serve the request — currently only reachable in stub mode (a live + * crawl), where `viewer_anchor_facts` can never exist (`buildViewerReadModel` + * refuses read-only accessors, and `viewer-build` refuses stub directories). + * A distinct `available: false` marker, rather than an empty + * `{ items: [], total: 0 }`, so the frontend can tell "not computed yet" + * apart from "genuinely zero inbound links" — the same distinction + * `/api/directory-tree`'s read-model-only `[]` return blurs. + */ +export interface InboundLinksUnavailable { + /** Always `false` — the discriminant frontend code checks for. */ + available: false; +} diff --git a/packages/@nitpicker/viewer/web/api/use-inbound-links-infinite.ts b/packages/@nitpicker/viewer/web/api/use-inbound-links-infinite.ts new file mode 100644 index 00000000..fb8b6d2a --- /dev/null +++ b/packages/@nitpicker/viewer/web/api/use-inbound-links-infinite.ts @@ -0,0 +1,48 @@ +import type { InfiniteQueryOptions } from './infinite-query-options.js'; +import type { InboundLinksResponse } from './use-inbound-links.js'; +import type { InboundLinkEntry } from '@nitpicker/query'; + +import { useInfiniteQuery } from '@tanstack/react-query'; + +import { apiGet } from './api-client.js'; +import { PAGE_SIZE } from './page-size.js'; + +/** One inbound-link row, as rendered by the inbound-links view. */ +export type InboundLinkRow = InboundLinkEntry; + +/** + * Infinite-scrolling inbound-link listing for one target page. Fetches + * `PAGE_SIZE` rows per request and advances via the server-issued + * `nextCursor` (keyset pagination) — `/api/pages/inbound-links` has no + * offset-vs-cursor branch to hide from this hook (unlike + * `useLinksInfinite`'s legacy fallback): it reads `viewer_anchor_facts` + * exclusively, so the cursor shape never changes between pages. + * + * In stub mode the endpoint responds `{ available: false }` with no + * `nextCursor` — `getNextPageParam` naturally reads that as "no next page" + * and the fetch stops after one page; the view checks `'available' in page` + * to render the stub-mode notice instead of an empty table. + * + * Retry is disabled (`retry: false`), matching `useInboundLinks`: a thrown + * error means the viewer read model is missing or stale, which does not + * resolve itself between retries. + * @param url - The target page's URL. + * @param options - Optional flags (`enabled`). + * @returns The TanStack infinite-query result. + */ +export function useInboundLinksInfinite(url: string, options?: InfiniteQueryOptions) { + return useInfiniteQuery({ + queryKey: ['inbound-links', url], + initialPageParam: null as string | null, + queryFn: ({ pageParam }) => + apiGet('/api/pages/inbound-links', { + url, + limit: PAGE_SIZE, + cursor: pageParam ?? undefined, + }), + getNextPageParam: (lastPage) => + ('available' in lastPage ? undefined : lastPage.nextCursor) ?? undefined, + enabled: (options?.enabled ?? true) && url !== '', + retry: false, + }); +} diff --git a/packages/@nitpicker/viewer/web/api/use-inbound-links.ts b/packages/@nitpicker/viewer/web/api/use-inbound-links.ts new file mode 100644 index 00000000..f454d63b --- /dev/null +++ b/packages/@nitpicker/viewer/web/api/use-inbound-links.ts @@ -0,0 +1,61 @@ +import type { InboundLinkList } from '@nitpicker/query'; +import type { QueryKey } from '@tanstack/react-query'; + +import { useQuery } from '@tanstack/react-query'; + +import { apiGet } from './api-client.js'; + +/** + * Response shape `/api/pages/inbound-links` returns instead of an + * {@link InboundLinkList} when the viewer read model cannot serve the + * request — currently only reachable in stub mode (a live crawl), where + * `viewer_anchor_facts` can never exist. Mirrors + * `register-inbound-links-route.ts`'s `InboundLinksUnavailable` on the wire; + * declared independently here rather than imported, since `web/` never + * imports types from the backend's `src/` — the two are built by separate + * toolchains (Vite vs `tsc`) despite sharing a package. + */ +export interface InboundLinksUnavailable { + /** Always `false` — the discriminant this hook's callers check for. */ + available: false; +} + +/** The union `useInboundLinks`/`useInboundLinksInfinite` callers must narrow before reading `items`/`total`. */ +export type InboundLinksResponse = InboundLinkList | InboundLinksUnavailable; + +/** + * Fetches one bounded window of a target page's inbound links — MPA-mode + * paging (`limit`/`offset`) or a count-only read (`limit: 0`, e.g. Page + * Detail's referrer count). + * + * Retry is disabled (`retry: false`): a thrown error here means the viewer + * read model is missing or stale (see `requireViewerReadModel`), which does + * not resolve itself between retries — surfacing it immediately gives the + * `viewer-build` guidance to the user without a multi-retry delay. + * @param url - The target page's URL (query disabled when empty). + * @param params - `limit`/`offset` for this window. + * @param params.limit + * @param params.offset + * @param queryKey - The TanStack Query cache key. Must include every value + * that influences the response (`url`, `limit`, `offset`) so two requests + * never collapse onto the same cache entry. + * @param options - Optional flags (`enabled`). + * @param options.enabled + * @returns The TanStack Query result — narrow on `'available' in data` before + * reading `items`/`total`. + */ +export function useInboundLinks( + url: string, + params: { limit?: number; offset?: number }, + queryKey: QueryKey, + options?: { enabled?: boolean }, +) { + return useQuery({ + queryKey, + queryFn: () => + apiGet('/api/pages/inbound-links', { url, ...params }), + enabled: (options?.enabled ?? true) && url !== '', + retry: false, + placeholderData: (previous) => previous, + }); +} diff --git a/packages/@nitpicker/viewer/web/app.tsx b/packages/@nitpicker/viewer/web/app.tsx index 6cecbaa5..cdebdd24 100644 --- a/packages/@nitpicker/viewer/web/app.tsx +++ b/packages/@nitpicker/viewer/web/app.tsx @@ -14,6 +14,7 @@ import { ErrorsView } from './routes/errors-view.js'; import { ExternalLinksView } from './routes/external-links-view.js'; import { GraphView } from './routes/graph-view.js'; import { ImagesView } from './routes/images-view.js'; +import { InboundLinksView } from './routes/inbound-links-view.js'; import { IsolatedClustersView } from './routes/isolated-clusters-view.js'; import { IsolatedPagesView } from './routes/isolated-pages-view.js'; import { MismatchesView } from './routes/mismatches-view.js'; @@ -59,6 +60,7 @@ export function App() { } /> } /> } /> + } /> } /> } /> } /> diff --git a/packages/@nitpicker/viewer/web/i18n/translations.ts b/packages/@nitpicker/viewer/web/i18n/translations.ts index a2d208f5..8257d8d0 100644 --- a/packages/@nitpicker/viewer/web/i18n/translations.ts +++ b/packages/@nitpicker/viewer/web/i18n/translations.ts @@ -180,7 +180,7 @@ export const translations: Record> = { pageDetail: { title: 'Page detail', description: - 'Full metadata, inbound/outbound links, redirects, and the stored HTML snapshot for one page.', + 'Full metadata, outbound links, redirects, and the stored HTML snapshot for one page. Inbound links are shown by count, with a link to the full list.', skipReason: 'Skipped (reason)', status: 'Status', contentType: 'Content-Type', @@ -192,6 +192,7 @@ export const translations: Record> = { ogImage: 'og:image', outbound: 'Outbound links', inbound: 'Inbound links', + viewInboundLinks: 'View all inbound links', linksTruncated: 'Showing the first {max} of {total} entries.', redirectedFrom: 'Redirected from', consoleLogs: 'Console logs', @@ -215,6 +216,14 @@ export const translations: Record> = { loadingMainContent: 'Loading main content…', noMainContent: 'No main content data available.', }, + inboundLinks: { + title: 'Inbound links', + description: 'Every referrer page linking to one target page.', + colReferrer: 'Referrer URL', + colAnchorText: 'Anchor text', + colCount: 'Count', + unavailable: 'Available once the crawl finishes.', + }, resources: { title: 'Resources', description: @@ -578,7 +587,7 @@ export const translations: Record> = { pageDetail: { title: 'ページ詳細', description: - '1 ページの全メタデータ、内部・外部リンク、リダイレクト、保存された HTML スナップショットを表示します。', + '1 ページの全メタデータ、発リンク、リダイレクト、保存された HTML スナップショットを表示します。被リンクは件数のみ表示し、一覧は別画面へのリンクから確認できます。', skipReason: '除外理由', status: 'ステータス', contentType: 'Content-Type', @@ -590,6 +599,7 @@ export const translations: Record> = { ogImage: 'og:image', outbound: '発リンク', inbound: '被リンク', + viewInboundLinks: '被リンク一覧を見る', linksTruncated: '全 {total} 件のうち先頭 {max} 件のみ表示しています。', redirectedFrom: 'リダイレクト元', consoleLogs: 'コンソールログ', @@ -613,6 +623,14 @@ export const translations: Record> = { loadingMainContent: 'メインコンテンツを読み込み中…', noMainContent: 'メインコンテンツのデータがありません。', }, + inboundLinks: { + title: '被リンク', + description: '対象ページにリンクしているすべての参照元ページ。', + colReferrer: '参照元 URL', + colAnchorText: 'アンカーテキスト', + colCount: '件数', + unavailable: 'クロール完了後に利用できます。', + }, resources: { title: 'リソース', description: diff --git a/packages/@nitpicker/viewer/web/routes/inbound-links-view.tsx b/packages/@nitpicker/viewer/web/routes/inbound-links-view.tsx new file mode 100644 index 00000000..70740dd6 --- /dev/null +++ b/packages/@nitpicker/viewer/web/routes/inbound-links-view.tsx @@ -0,0 +1,141 @@ +import type { InboundLinkRow } from '../api/use-inbound-links-infinite.js'; +import type { InboundLinksResponse } from '../api/use-inbound-links.js'; +import type { ColumnDef } from '@tanstack/react-table'; + +import { useMemo } from 'react'; +import { Link, useSearchParams } from 'react-router'; + +import { useInboundLinksInfinite } from '../api/use-inbound-links-infinite.js'; +import { useInboundLinks } from '../api/use-inbound-links.js'; +import { DataTable } from '../components/data-table.js'; +import { ViewHeader } from '../components/view-header.js'; +import { useListPagination } from '../hooks/use-list-pagination.js'; +import { useI18n } from '../i18n/use-i18n.js'; + +/** + * Extracts the row window from a response that may be the stub-mode + * `{ available: false }` marker. + * @param response - The raw hook response, if loaded yet. + * @returns The row window, or `[]` while loading or unavailable. + */ +function toItems(response: InboundLinksResponse | undefined): InboundLinkRow[] { + return response && !('available' in response) ? response.items : []; +} + +/** + * Extracts the total count from a response that may be the stub-mode + * `{ available: false }` marker. + * @param response - The raw hook response, if loaded yet. + * @returns The total count, or `0` while loading or unavailable. + */ +function toTotal(response: InboundLinksResponse | undefined): number { + return response && !('available' in response) ? response.total : 0; +} + +/** + * The inbound-links view: every referrer page linking to one target page, + * with anchor text and per-referrer anchor count. Split out of Page Detail + * (issue #235) — a page's referrer count can reach the hundreds of + * thousands on a large site, too large to embed in a single-page response. + * The target URL comes from the `url` query param, same as Page Detail. + * @returns The inbound-links view element. + */ +export function InboundLinksView() { + const [params] = useSearchParams(); + const { t } = useI18n(); + const url = params.get('url') ?? ''; + const { mode, pageSize, currentPage, setPage, setPageSize } = useListPagination(); + + const offset = (currentPage - 1) * pageSize; + const paged = useInboundLinks( + url, + { limit: pageSize, offset }, + ['inbound-links-paged', url, pageSize, currentPage], + { enabled: mode === 'mpa' }, + ); + const infinite = useInboundLinksInfinite(url, { enabled: mode === 'virtual' }); + const infiniteRows = useMemo( + () => infinite.data?.pages.flatMap((page) => toItems(page)) ?? [], + [infinite.data], + ); + + const columns = useMemo[]>( + () => [ + { + id: 'url', + header: t('views.inboundLinks.colReferrer'), + size: 420, + accessorFn: (r) => r.url, + }, + { + id: 'textContent', + header: t('views.inboundLinks.colAnchorText'), + size: 260, + accessorFn: (r) => r.textContent ?? '—', + }, + { + id: 'count', + header: t('views.inboundLinks.colCount'), + size: 90, + accessorFn: (r) => r.count, + }, + ], + [t], + ); + + if (!url) { + return
{t('views.pageDetail.noPage')}
; + } + + const currentResponse = mode === 'mpa' ? paged.data : infinite.data?.pages[0]; + const isUnavailable = currentResponse != null && 'available' in currentResponse; + + return ( +
+ + + {t('common.back')} {t('views.pageDetail.title')} + +
+
URL
+
{url}
+
+ {isUnavailable ? ( +

{t('views.inboundLinks.unavailable')}

+ ) : mode === 'mpa' ? ( + + ) : ( + { + void infinite.fetchNextPage(); + }} + /> + )} +
+ ); +} diff --git a/packages/@nitpicker/viewer/web/routes/page-detail-view.tsx b/packages/@nitpicker/viewer/web/routes/page-detail-view.tsx index b59f91b8..feef64ff 100644 --- a/packages/@nitpicker/viewer/web/routes/page-detail-view.tsx +++ b/packages/@nitpicker/viewer/web/routes/page-detail-view.tsx @@ -1,5 +1,6 @@ import { Link, useSearchParams } from 'react-router'; +import { useInboundLinks } from '../api/use-inbound-links.js'; import { usePageDetail } from '../api/use-page-detail.js'; import { usePageHtml } from '../api/use-page-html.js'; import { usePageMainContents } from '../api/use-page-main-contents.js'; @@ -10,8 +11,11 @@ import { useI18n } from '../i18n/use-i18n.js'; const MAX_LINKS_DISPLAYED = 200; /** - * Full detail for a single page: metadata, inbound/outbound links, redirects, - * and the stored HTML snapshot. The target URL comes from the `url` query param. + * Full detail for a single page: metadata, outbound links, redirects, and + * the stored HTML snapshot. Inbound links are summarized by count with a + * link to the dedicated `/pages/inbound-links` list (issue #235) — a page's + * referrer count can reach the hundreds of thousands on a large site, too + * large to embed here. The target URL comes from the `url` query param. * @returns The page detail view element. */ export function PageDetailView() { @@ -26,6 +30,19 @@ export function PageDetailView() { // `data` is still undefined and its eventual `isExternal` value is unknown. const html = usePageHtml(data && !data.isExternal ? url : ''); const mainContents = usePageMainContents(data && !data.isExternal ? url : ''); + // Count-only read (`limit: 0`) — the full inbound-link window lives at + // `/pages/inbound-links`, not here (see this component's docs). + const { + data: inboundData, + isLoading: inboundIsLoading, + isError: inboundIsError, + error: inboundError, + } = useInboundLinks(url, { limit: 0 }, ['inbound-links-count', url], { + enabled: url !== '', + }); + const inboundUnavailable = inboundData != null && 'available' in inboundData; + const inboundTotal = + inboundData && !('available' in inboundData) ? inboundData.total : null; if (!url) { return
{t('views.pageDetail.noPage')}
; @@ -158,24 +175,22 @@ export function PageDetailView() {

- {t('views.pageDetail.inbound')} ({data.inboundLinks.length}) + {t('views.pageDetail.inbound')} + {inboundTotal == null ? '' : ` (${inboundTotal})`}

-
    - {data.inboundLinks.slice(0, MAX_LINKS_DISPLAYED).map((link, index) => ( -
  • - - {link.url} - -
    • - ))} -
- {data.inboundLinks.length > MAX_LINKS_DISPLAYED && ( -

- {t('views.pageDetail.linksTruncated', { - max: MAX_LINKS_DISPLAYED, - total: data.inboundLinks.length, - })} -

+ {inboundUnavailable ? ( +

{t('views.inboundLinks.unavailable')}

+ ) : inboundIsError ? ( +

{inboundError.message}

+ ) : inboundIsLoading ? ( +

{t('common.loading')}

+ ) : ( + inboundTotal != null && + inboundTotal > 0 && ( + + {t('views.pageDetail.viewInboundLinks')} + + ) )} {data.outboundLinks.length > 0 && (