Skip to content

fix(crawler): stop self-generating pagination URLs, add opt-in same-cluster soft cap (#208) #247

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
YusukeHirao merged 10 commits into from
Jul 31, 2026
Merged

fix(crawler): stop self-generating pagination URLs, add opt-in same-cluster soft cap (#208) #247

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
7131d47
chore(repo): register dedupe-cap e2e shard and cspell dictionary terms
YusukeHirao Jul 30, 2026
f3d9fb8
feat(crawler): fix self-generating pagination URLs and add dedupe-cap…
YusukeHirao Jul 30, 2026
7c5fd9a
feat(query): add duplicate-body-cluster and dedupe-cap-event queries
YusukeHirao Jul 30, 2026
e41a423
feat(mcp-server): add find_duplicate_clusters and list_dedupe_cap_eve…
YusukeHirao Jul 30, 2026
7cb87f1
feat(viewer): add Duplicate Clusters view for issue #208
YusukeHirao Jul 30, 2026
ef46282
feat(cli): add --dedupe-cap flags and duplicate-clusters/dedupe-cap-e…
YusukeHirao Jul 30, 2026
5331fca
test(test-server): add dedupe-cap e2e trap fixture
YusukeHirao Jul 30, 2026
95072de
docs(repo): document issue #208 changes in ARCHITECTURE.md
YusukeHirao Jul 30, 2026
e8dcf76
fix(crawler): fix rejected_count finalization gap, stale body_hash co…
YusukeHirao Jul 30, 2026
112c6cd
fix(query): validate limit/offset and batch duplicate-cluster URL fet…
YusukeHirao Jul 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/e2e.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,7 @@ jobs:
config-persistence.e2e.ts,
scope-auth-leak.e2e.ts,
main-contents.e2e.ts,
dedupe-cap.e2e.ts,
]
- id: retry-exclude
files:
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/viewer-e2e.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,3 +45,4 @@ jobs:
- run: yarn workspace @nitpicker/viewer test:e2e:directory-tree
- run: yarn workspace @nitpicker/viewer test:e2e:template-clusters
- run: yarn workspace @nitpicker/viewer test:e2e:inbound-links
- run: yarn workspace @nitpicker/viewer test:e2e:duplicate-clusters
20 changes: 19 additions & 1 deletion ARCHITECTURE.md
View file
Open in desktop

Large diffs are not rendered by default.

5 changes: 5 additions & 0 deletions cspell.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,10 @@
"bench-output/**"
],
"words": [
// Misra-Gries algorithm (DedupeCapTracker, issue #208)
"Misra",
"Gries",

// nitpicker
"readtext",
"unanalyzed",
Expand All @@ -32,6 +36,7 @@
"greppable",
"desync",
"lossily",
"unfinalized",

// HTML
"noarchive",
Expand Down
60 changes: 31 additions & 29 deletions packages/@nitpicker/cli/docs/crawl.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -166,35 +166,37 @@ npx @nitpicker/cli crawl --diff ./before.nitpicker ./after.nitpicker

## オプション一覧

| オプション | 型 | 説明 |
| ------------------------------------------ | ------------------ | ----------------------------------------------------- |
| `--resume`, `-R` | string | stubディレクトリからクロールを再開 |
| `--append`, `-A` | string, repeatable | 既存アーカイブへ新しい再帰クロール起点を追加 |
| `--retry-failed` | boolean | 既存アーカイブ内の失敗ページを再取得 |
| `--inventory` | string | サーバー側URLリストを既存アーカイブへ取り込み |
| `--interval`, `-I` | number | リクエスト間隔をミリ秒で指定 |
| `--image` / `--no-image` | boolean | 画像を取得するか。既定は有効 |
| `--fetch-external` / `--no-fetch-external` | boolean | 外部リンクを取得するか。既定は有効 |
| `--parallels`, `-P` | number | 並列スクレイピング数 |
| `--recursive` / `--no-recursive` | boolean | 再帰クロールするか。既定は有効 |
| `--exclude` | string, repeatable | 除外するページURLパスのglob |
| `--exclude-keyword` | string, repeatable | ページ本文に含まれる除外キーワード |
| `--exclude-url` | string, repeatable | 除外する外部URL prefix |
| `--disable-queries`, `-Q` | boolean | URLのクエリ文字列を無効化 |
| `--image-file-size-threshold` | number | 画像ファイルサイズのしきい値 |
| `--single` | boolean | 単一ページモード |
| `--max-excluded-depth` | number | 指定深さを超えるクロールを避ける |
| `--retry` | number | URLごとのスクレイプ失敗リトライ回数。既定は `3` |
| `--list` | string, repeatable | 指定URLリストだけをクロール |
| `--list-file` | string | URLリストファイルだけをクロール |
| `--user-agent` | string | HTTPリクエストのUser-Agent |
| `--ignore-robots` | boolean | robots.txt制限を無視 |
| `--main-content-selector` | string | メインコンテンツ領域の自動検出を上書きするCSSセレクタ |
| `--output`, `-o` | string | 出力 `.nitpicker` ファイルパス |
| `--strict` | boolean | 外部リンクエラーを致命的エラーとして扱う |
| `--verbose` | boolean | 詳細ログを出力 |
| `--silent` | boolean | 標準出力ログを抑制 |
| `--diff` | boolean | 2つのアーカイブの差分を出力 |
| オプション | 型 | 説明 |
| ------------------------------------------ | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--resume`, `-R` | string | stubディレクトリからクロールを再開 |
| `--append`, `-A` | string, repeatable | 既存アーカイブへ新しい再帰クロール起点を追加 |
| `--retry-failed` | boolean | 既存アーカイブ内の失敗ページを再取得 |
| `--inventory` | string | サーバー側URLリストを既存アーカイブへ取り込み |
| `--interval`, `-I` | number | リクエスト間隔をミリ秒で指定 |
| `--image` / `--no-image` | boolean | 画像を取得するか。既定は有効 |
| `--fetch-external` / `--no-fetch-external` | boolean | 外部リンクを取得するか。既定は有効 |
| `--parallels`, `-P` | number | 並列スクレイピング数 |
| `--recursive` / `--no-recursive` | boolean | 再帰クロールするか。既定は有効 |
| `--exclude` | string, repeatable | 除外するページURLパスのglob |
| `--exclude-keyword` | string, repeatable | ページ本文に含まれる除外キーワード |
| `--exclude-url` | string, repeatable | 除外する外部URL prefix |
| `--disable-queries`, `-Q` | boolean | URLのクエリ文字列を無効化 |
| `--image-file-size-threshold` | number | 画像ファイルサイズのしきい値 |
| `--single` | boolean | 単一ページモード |
| `--max-excluded-depth` | number | 指定深さを超えるクロールを避ける |
| `--retry` | number | URLごとのスクレイプ失敗リトライ回数。既定は `3` |
| `--list` | string, repeatable | 指定URLリストだけをクロール |
| `--list-file` | string | URLリストファイルだけをクロール |
| `--user-agent` | string | HTTPリクエストのUser-Agent |
| `--ignore-robots` | boolean | robots.txt制限を無視 |
| `--main-content-selector` | string | メインコンテンツ領域の自動検出を上書きするCSSセレクタ |
| `--output`, `-o` | string | 出力 `.nitpicker` ファイルパス |
| `--strict` | boolean | 外部リンクエラーを致命的エラーとして扱う |
| `--verbose` | boolean | 詳細ログを出力 |
| `--silent` | boolean | 標準出力ログを抑制 |
| `--diff` | boolean | 2つのアーカイブの差分を出力 |
| `--dedupe-cap` | number | 同一クラスタ soft cap。URL形状（例: `/news/date/{n}/`）ごとにtitle/description/og:tagが一致する観測がこの件数に達したら以降の新規URLをenqueueしない。opt-in（省略で無効）。自己生成型のpager/queryパラメータtrapへの保険。発火内容は `query dedupe-cap-events` で確認可能 |
| `--dedupe-map-cap` | number | `--dedupe-cap` が同時追跡するURL形状の数の上限。超過分は最も長く未更新の形状から破棄。`--dedupe-cap` 指定時のみ有効 |

## 終了コード

Expand Down
2 changes: 2 additions & 0 deletions packages/@nitpicker/cli/docs/query.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,8 @@ npx @nitpicker/cli query ./site.nitpicker page-detail --url https://example.com/
| `images` | 画像一覧と画像品質フィルタ |
| `violations` | 分析プラグインの違反結果 |
| `duplicates` | title/descriptionの重複 |
| `duplicate-clusters` | 同一body_hashクラスタの集約（trap兆候でソート） |
| `dedupe-cap-events` | `--dedupe-cap` の同一クラスタ soft cap 発火履歴 |
| `mismatches` | canonical/OGPメタデータの不一致 |
| `headers` | セキュリティヘッダー確認 |
| `resource-referrers` | 指定リソースの参照元ページ |
Expand Down
8 changes: 8 additions & 0 deletions packages/@nitpicker/cli/src/commands/crawl.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,14 @@ export const commandDef = {
type: 'boolean',
desc: 'Diff mode',
},
dedupeCap: {
type: 'number',
desc: 'Same-cluster soft cap: stop enqueueing newly-discovered internal URLs whose shape (e.g. `/news/date/{n}/`) has accumulated this many matching-title/description/og-tag observations. Opt-in — omit to disable. Backstop against a site that keeps serving 2xx for a self-generating pager/query-parameter trap; see `query dedupe-cap-events` for what fired.',
},
dedupeMapCap: {
type: 'number',
desc: 'Hard cap on the number of distinct URL shapes --dedupe-cap tracks at once; the least-recently-touched shape is evicted beyond this. Only relevant when --dedupe-cap is set.',
},
},
} as const satisfies CommandDef;

Expand Down
19 changes: 19 additions & 0 deletions packages/@nitpicker/cli/src/commands/pipeline.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,25 @@ describe('pipeline command', () => {
);
});

it("forwards --dedupe-cap/--dedupe-map-cap to startCrawl (pipeline.ts hand-writes its own flags object rather than reusing crawl.ts's mapper, see the TODO on commandDef.flags)", async () => {
vi.mocked(startCrawlFn).mockResolvedValue('/tmp/site.nitpicker');
vi.mocked(analyzeFn).mockResolvedValue();

await pipeline(['https://example.com'], {
...defaultFlags,
dedupeCap: 5,
dedupeMapCap: 2000,
});

expect(startCrawlFn).toHaveBeenCalledWith(
['https://example.com'],
expect.objectContaining({
dedupeCap: 5,
dedupeMapCap: 2000,
}),
);
});

it('runs crawl, analyze, and report when --sheet is provided', async () => {
const sheetUrl = 'https://docs.google.com/spreadsheets/d/xxx';
vi.mocked(startCrawlFn).mockResolvedValue('/tmp/site.nitpicker');
Expand Down
10 changes: 10 additions & 0 deletions packages/@nitpicker/cli/src/commands/pipeline.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,14 @@ export const commandDef = {
type: 'boolean',
desc: 'Treat external link errors as fatal (exit code 1 instead of 2)',
},
dedupeCap: {
type: 'number',
desc: 'Same-cluster soft cap: stop enqueueing newly-discovered internal URLs whose shape (e.g. `/news/date/{n}/`) has accumulated this many matching-title/description/og-tag observations. Opt-in — omit to disable. Backstop against a site that keeps serving 2xx for a self-generating pager/query-parameter trap; see `query dedupe-cap-events` for what fired.',
},
dedupeMapCap: {
type: 'number',
desc: 'Hard cap on the number of distinct URL shapes --dedupe-cap tracks at once; the least-recently-touched shape is evicted beyond this. Only relevant when --dedupe-cap is set.',
},
// analyze flags
all: {
type: 'boolean',
Expand Down Expand Up @@ -248,6 +256,8 @@ export async function pipeline(args: string[], flags: PipelineFlags) {
retryFailed: false,
inventory: undefined,
diff: undefined,
dedupeCap: flags.dedupeCap,
dedupeMapCap: flags.dedupeMapCap,
});
} catch (error) {
if (
Expand Down
6 changes: 5 additions & 1 deletion packages/@nitpicker/cli/src/commands/query.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,11 @@ export const commandDef = {
},
pagesLimit: {
type: 'number',
desc: 'Inline member-page URL sample size per duplicate group (duplicates). Defaults to 20.',
desc: 'Inline member-page URL sample size per duplicate group (duplicates), or per body-hash cluster (duplicate-clusters). Defaults to 20.',
},
minCount: {
type: 'number',
desc: 'Minimum cluster size to include (duplicate-clusters). Defaults to 10.',
},
url: {
type: 'string',
Expand Down
Loading
Loading