Skip to content

test: mask PDF region in visual regression snapshots - #448

Merged
tomusdrw merged 2 commits into
mainfrom
fix/mask-pdf-in-snapshot-tests
Jun 22, 2026
Merged

tomusdrw merged 2 commits into
mainfrom
fix/mask-pdf-in-snapshot-tests

Conversation

@tomusdrw

@tomusdrw tomusdrw commented Jun 22, 2026 •

Copy link
Copy Markdown
Member

Problem

The visual regression suite fails intermittently because the PDF.js canvas occasionally renders a few pixels off between otherwise identical runs. A positional shift is the worst case for pixel-diff thresholds: it moves every glyph edge, changing a band of pixels around all text, which reliably trips the existing maxDiffPixelRatio: 0.02 threshold.

Approach

Rather than loosening the global threshold (which would also blind us to genuine layout regressions of the same magnitude — and these are layout tests), this masks the PDF viewer out of the layout/chrome snapshots. The surrounding UI keeps the tight 2% tolerance; the inherently non-deterministic PDF region is ignored.

Playwright paints a solid magenta box over the masked element's bounding box. The jitter lives in the canvas content, not in the position of .pdf-viewer-root (an absolute-positioned container that fills its pane), so the masked comparison is deterministic.

Changes

  • New helper tools/snapshot-tests/tests/utils/screenshot-opts.ts — maskedPdfScreenshot(page) masks .pdf-viewer-root (matches both the main and split-pane viewers)
  • Applied to all full-page snapshots in basic-snapshots, split-screen, and narrow-screen specs
  • The clip-based notes-tab shots already exclude the PDF and were left untouched
  • Regenerated the 50 affected linux baselines via npm run docker:test:update

Example regenerated baseline (homepage outline) — PDF area masked, all chrome still verified:

⚠️ Notes for reviewer

  • darwin baselines are now stale. Only the linux baselines (used by CI) were regenerated. Anyone running the suite locally on macOS will need cd tools/snapshot-tests && npm run test:local -- --update-snapshots, or we can regenerate+commit them in a follow-up.
  • Coverage trade-off: these tests no longer detect PDF render regressions (wrong page, blank canvas, broken scale) — only that it loads (data-pdf-ready). A dedicated PDF-render test with its own looser tolerance could restore that coverage if desired.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Tests
    • Improved visual regression stability by standardizing screenshot assertions to use a masking-based helper for PDF viewer content.
    • Updated snapshot specs covering basic, narrow, and split/compare views to rely on consistent screenshot options rather than shared full-page settings.

@netlify

netlify Bot commented Jun 22, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for graypaper-reader ready!

Name Link
🔨 Latest commit 5dd5e69
🔍 Latest deploy log https://app.netlify.com/projects/graypaper-reader/deploys/6a390aed77a2fa00083a4612
😎 Deploy Preview https://deploy-preview-448--graypaper-reader.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@coderabbitai

coderabbitai Bot commented Jun 22, 2026 •

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 9fd12fd2-da03-4ace-bd1b-554bbfa65764

📥 Commits

Reviewing files that changed from the base of the PR and between ba790b1 and 552cb3a.

⛔ Files ignored due to path filters (52)
  • tools/snapshot-tests/tests/basic-snapshots.spec.ts-snapshots/homepage-outline-dark-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/basic-snapshots.spec.ts-snapshots/homepage-outline-light-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/basic-snapshots.spec.ts-snapshots/notes-tab-after-note-activation-dark-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/basic-snapshots.spec.ts-snapshots/notes-tab-after-note-activation-light-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/basic-snapshots.spec.ts-snapshots/search-initial-dark-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/basic-snapshots.spec.ts-snapshots/search-initial-light-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/basic-snapshots.spec.ts-snapshots/search-with-query-dark-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/basic-snapshots.spec.ts-snapshots/search-with-query-light-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/basic-snapshots.spec.ts-snapshots/search-with-results-dark-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/basic-snapshots.spec.ts-snapshots/search-with-results-light-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/basic-snapshots.spec.ts-snapshots/topbar-github-expanded-dark-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/basic-snapshots.spec.ts-snapshots/topbar-github-expanded-light-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/basic-snapshots.spec.ts-snapshots/topbar-github-option-hover-dark-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/basic-snapshots.spec.ts-snapshots/topbar-github-option-hover-light-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/basic-snapshots.spec.ts-snapshots/topbar-more-clicked-dark-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/basic-snapshots.spec.ts-snapshots/topbar-more-clicked-light-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/basic-snapshots.spec.ts-snapshots/topbar-more-settings-clicked-dark-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/basic-snapshots.spec.ts-snapshots/topbar-more-settings-clicked-light-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/basic-snapshots.spec.ts-snapshots/topbar-notes-dropdown-dark-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/basic-snapshots.spec.ts-snapshots/topbar-notes-dropdown-light-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/basic-snapshots.spec.ts-snapshots/topbar-notes-selected-off-dark-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/basic-snapshots.spec.ts-snapshots/topbar-notes-selected-off-light-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/basic-snapshots.spec.ts-snapshots/topbar-notes-selected-on-dark-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/basic-snapshots.spec.ts-snapshots/topbar-notes-selected-on-light-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/narrow-screen.spec.ts-snapshots/narrow-layout-collapsed-dark-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/narrow-screen.spec.ts-snapshots/narrow-layout-collapsed-light-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/narrow-screen.spec.ts-snapshots/narrow-layout-expanded-dark-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/narrow-screen.spec.ts-snapshots/narrow-layout-expanded-light-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/narrow-screen.spec.ts-snapshots/narrow-layout-ignores-split-dark-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/narrow-screen.spec.ts-snapshots/narrow-layout-ignores-split-light-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/narrow-screen.spec.ts-snapshots/narrow-layout-re-collapsed-dark-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/narrow-screen.spec.ts-snapshots/narrow-layout-re-collapsed-light-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/narrow-screen.spec.ts-snapshots/tablet-layout-dark-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/narrow-screen.spec.ts-snapshots/tablet-layout-light-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/split-screen.spec.ts-snapshots/sidebar-with-split-tab-dark-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/split-screen.spec.ts-snapshots/sidebar-with-split-tab-light-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/split-screen.spec.ts-snapshots/split-header-options-menu-dark-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/split-screen.spec.ts-snapshots/split-header-options-menu-light-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/split-screen.spec.ts-snapshots/split-header-version-dropdown-dark-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/split-screen.spec.ts-snapshots/split-header-version-dropdown-light-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/split-screen.spec.ts-snapshots/split-sidebar-overlay-closed-dark-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/split-screen.spec.ts-snapshots/split-sidebar-overlay-closed-light-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/split-screen.spec.ts-snapshots/split-sidebar-overlay-dark-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/split-screen.spec.ts-snapshots/split-sidebar-overlay-light-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/split-screen.spec.ts-snapshots/split-view-closed-dark-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/split-screen.spec.ts-snapshots/split-view-closed-light-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/split-screen.spec.ts-snapshots/split-view-opened-dark-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/split-screen.spec.ts-snapshots/split-view-opened-light-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/split-screen.spec.ts-snapshots/split-view-via-url-dark-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/split-screen.spec.ts-snapshots/split-view-via-url-light-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/split-screen.spec.ts-snapshots/version-dropdown-with-compare-dark-mode-linux.png is excluded by !**/*.png
  • tools/snapshot-tests/tests/split-screen.spec.ts-snapshots/version-dropdown-with-compare-light-mode-linux.png is excluded by !**/*.png
📒 Files selected for processing (4)
  • tools/snapshot-tests/tests/basic-snapshots.spec.ts
  • tools/snapshot-tests/tests/narrow-screen.spec.ts
  • tools/snapshot-tests/tests/split-screen.spec.ts
  • tools/snapshot-tests/tests/utils/screenshot-opts.ts
🚧 Files skipped from review as they are similar to previous changes (4)
  • tools/snapshot-tests/tests/utils/screenshot-opts.ts
  • tools/snapshot-tests/tests/narrow-screen.spec.ts
  • tools/snapshot-tests/tests/split-screen.spec.ts
  • tools/snapshot-tests/tests/basic-snapshots.spec.ts

📝 Walkthrough

Walkthrough

A new maskedPdfScreenshot(page) helper is added in tools/snapshot-tests/tests/utils/screenshot-opts.ts. It returns fullPage: true plus a Playwright mask over .pdf-viewer-root. All three snapshot test specs (basic-snapshots, narrow-screen, split-screen) drop their local screenshotOpts constants and adopt this helper uniformly.

Changes

PDF Viewer Masking in Snapshot Tests

Layer / File(s) Summary
maskedPdfScreenshot utility
tools/snapshot-tests/tests/utils/screenshot-opts.ts
New exported function returning { fullPage: true, mask: [page.locator('.pdf-viewer-root')] } with inline docs explaining the PDF.js canvas non-determinism rationale.
Adoption across all three spec files
tools/snapshot-tests/tests/basic-snapshots.spec.ts, tools/snapshot-tests/tests/narrow-screen.spec.ts, tools/snapshot-tests/tests/split-screen.spec.ts
Adds maskedPdfScreenshot import, removes per-file screenshotOpts constants, and replaces all toHaveScreenshot option arguments with maskedPdfScreenshot(page) across every snapshot assertion in all three suites.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

Possibly related PRs

  • FluffyLabs/graypaper-reader#378: Modifies the same narrow-screen.spec.ts and split-screen.spec.ts snapshot configs, switching between screenshot option approaches.

Poem

🐇 A mask upon the PDF's face,
So flaky pixels find no place.
The canvas shifts, but we don't care —
We hide it all with pixel flair!
Now screenshots pass without a trace. ✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the primary change: introducing masking for the PDF region in snapshot tests to address rendering flakiness.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/mask-pdf-in-snapshot-tests

Comment @coderabbitai help to get the list of available commands and usage tips.

The PDF.js canvas occasionally renders a few pixels off between otherwise
identical runs. Because a positional shift moves every glyph edge, it changes
a band of pixels around all text and reliably trips the 2% pixel-diff
threshold, making the snapshot suite flaky.

Rather than loosening the global threshold (which would also blind us to real
layout regressions of the same magnitude), mask the PDF viewer(s) out of the
layout/chrome snapshots. The surrounding UI keeps the tight tolerance; the
inherently non-deterministic PDF region is ignored.

- Add maskedPdfScreenshot(page) helper masking .pdf-viewer-root (matches both
  the main viewer and the split-pane viewer)
- Apply it to all full-page snapshots in basic/split-screen/narrow-screen specs
  (clip-based notes-tab shots already exclude the PDF, left untouched)
- Regenerate the linux baselines

Trade-off: these tests no longer detect PDF *render* regressions (wrong page,
blank canvas, broken scale) — only that it loads. A dedicated PDF-render test
with its own looser tolerance could restore that coverage if wanted.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@tomusdrw
tomusdrw force-pushed the fix/mask-pdf-in-snapshot-tests branch from ba790b1 to 552cb3a Compare June 22, 2026 10:09
@github-actions

Copy link
Copy Markdown
Contributor

Visual Regression Test Report ✅ Passed

Github run id: 27945268230

🔗 Artifacts: Download

@tomusdrw
tomusdrw merged commit 4137f11 into main Jun 22, 2026
5 checks passed
@tomusdrw
tomusdrw deleted the fix/mask-pdf-in-snapshot-tests branch June 22, 2026 10:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant