feat: surface page-cluster's ClusterReason on templateKey classification - #245
Merged
Conversation
…ess fix tools#932 fixed onClusterReason forcing the small-corpus in-memory path even when onProgress was also passed, freezing the UI on archives at or below the 20,000-page inline threshold. 0.5.3 lets onProgress and onClusterReason compose independently.
Update the template classification section to cover the new page_template_clusters table, classifyPageTemplates's onClusterReason wiring, and the compute-css-intersection.ts / distinctiveStylesheetUrls split. Registers "Jaccard" in cspell for the new query-layer JSDoc.
Adds page_template_clusters (zstd-compressed JSON BLOB, one row per template key) alongside the existing page_templates table, and extends replacePageTemplates with an optional clusterReasonsByTemplateKey param. Compression is factored into a shared compressPayload primitive reused by upsert-json-ref.ts and write-page-html-blob.ts. each-splitted now accepts readonly arrays.
classifyPageTemplates now always passes onClusterReason to
resolvePageClusterKeys (page-cluster 0.5.3+ composes it with onProgress
without demoting off the progress-emitting path) and returns
{ templateKeysByUrl, clusterReasonsByTemplateKey } instead of a bare
Map. The persistent cache and nitpicker.ts's Phase 3 call site are
updated to match.
…aries Adds has-page-template-clusters-table.ts, is-template-cluster-reason.ts (a fail-closed shape guard covering nested blocking/landmarks entries), load-template-cluster-reasons.ts (filtered to the template keys the caller actually needs), and summarize-template-cluster-reason.ts (trims large token/shell-token arrays for API transport). listPageTemplateClusters now attaches TemplateClusterSummary.reason. compute-css-intersection.ts's JSDoc is corrected: build-cluster-reason is public since page-cluster 0.5.2, so the two metrics now coexist rather than one being a stand-in for the other.
…s view clusterHeading now prioritizes the reason's distinctive stylesheet filenames over the raw common-stylesheet intersection, appending the top directory to disambiguate sibling clusters that share the same blocking-group CSS set. A new ClusterReasonSection renders blocking evidence, common DOM-structure tokens, common landmarks, and sibling cluster links, or a "not captured" notice when no reason was saved. get-blocking-kind-label.ts / get-landmark-type-label.ts follow the existing views.<enum>.<value>-plus-fallback pattern. The e2e fixture gains a sibling-cluster pair to exercise the disambiguation and Siblings-section link paths.
Reconciles this branch's page-cluster 0.5.3-based cluster-reason implementation with dev's independently merged (and inferior, 0.5.1-based) duplicate of the same feature (issue #234, closed by dev before this branch's implementation was ready). Kept this branch's design throughout: compressed single-BLOB `page_template_clusters` schema (vs dev's uncompressed per-column table), the `commonStylesheetFileNames` CSS-intersection fallback dev's version had dropped, and the async, non-freezing `onClusterReason` wiring only possible once page-cluster 0.5.3 fixed the progress-freeze bug (tools#932). Salvaged dev's genuinely new, unrelated work as-is (inbound-links query/viewer feature) and folded dev's orphanMerge test case into this branch's list-page-template-clusters test suite.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
@d-zero/page-cluster'sClusterReason(why a group of pages was classified as the same template) alongside the existingtemplateKeyclassification, end to end: crawler storage → core classification → query summary → viewer UI.@d-zero/page-cluster@0.5.3, which fixes two upstream bugs found while implementing this (both filed and resolved before this PR): a missingexportsentry forbuild-cluster-reason(tools#930), andonClusterReasonforcing small corpora onto a synchronous, non-yielding classification path that froze the UI (tools#932).devhad already picked up with an independent,page-cluster@0.5.1-based implementation (merged and closed before this branch's implementation was ready). This PR reconciles both: it supersedes dev's version with this branch's design — a compressed single-BLOBpage_template_clusterstable (vs. dev's uncompressed per-column table), thecommonStylesheetFileNamesCSS-intersection fallback dev's version had dropped, and cluster-reason capture that doesn't freeze the UI (only possible with 0.5.3) — while keeping dev's unrelated inbound-links feature and folding dev'sorphanMergetest case into this branch's test suite.Changes
crawler: newpage_template_clusterstable (zstd-compressedreason_jsonBLOB),replacePageTemplatesgains an optionalclusterReasonsByTemplateKeyparam.core:classifyPageTemplatesalways requestsonClusterReasonand returnsclusterReasonsByTemplateKeyalongsidetemplateKeysByUrl; persistent cache entry shape extended (old-shape entries fall back to recompute).query:has-page-template-clusters-table,is-template-cluster-reason(shape validation, fail-closed),load-template-cluster-reasons,summarize-template-cluster-reason(trims verbatim data for the API);TemplateClusterSummary.reasonadded.viewer: template-clusters view shows blocking evidence, structural-core tokens, landmark profiles, and sibling-cluster links when a reason is available; falls back to a "not saved, re-run analyze --templates" hint otherwise.ARCHITECTURE.mdupdated for the new table and data flow.Test plan
NX_WORKSPACE_ROOT_PATH=<worktree> yarn build— 13/13 projectsyarn test— 569 files / 4012 tests passed, 4 skippedyarn lint/yarn lint:eslint:check— 0 errorsyarn workspace @nitpicker/viewer test:e2e— 52/52 passedyarn workspace @nitpicker/viewer test:e2e:template-clusters— 9/9 passed/code-review medium→/qa-engineer→/product-managerfindings all fixed🤖 Generated with Claude Code