fix: detect clickable elements inside cross-origin iframes#1423
Open
simonellefsen wants to merge 4 commits into
Open
fix: detect clickable elements inside cross-origin iframes#1423simonellefsen wants to merge 4 commits into
simonellefsen wants to merge 4 commits into
Conversation
find_cursor_interactive_elements was always called with the parent session_id. For cross-origin iframes the JavaScript therefore ran in the main frame's document and found none of the iframe's elements, so every clickable <div> inside such an iframe (e.g. game buttons rendered with onclick / cursor:pointer) appeared as plain StaticText without a ref. Switch to effective_session_id, which is the dedicated CDP session that Target.setAutoAttach attaches for cross-origin iframes. When there is no iframe in play effective_session_id == session_id, so the main-frame behaviour is unchanged. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
|
Someone is attempting to deploy a commit to the Vercel Labs Team on Vercel. A member of the Team first needs to authorize it. |
- Extract ref-assignment predicate into `should_assign_ref` helper so it can be tested without a live CDP connection. - `test_should_assign_ref_*`: verify interactive roles always get refs, content roles require a name, generic divs without cursor info are skipped, and generic divs WITH a matching backendNodeId in cursor_elements DO get refs (the core case for game buttons in iframes). - `test_cross_origin_iframe_effective_session_differs_from_parent`: regression guard — documents that effective_session_id must be passed to find_cursor_interactive_elements for cross-origin iframes. - `test_main_frame_effective_session_equals_parent` and `test_same_origin_iframe_effective_session_equals_parent`: confirm the fix is a no-op for main-frame and same-origin iframe snapshots. 25 snapshot tests pass (was 16). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…=none) Chrome completely omits elements with role="none/presentation" from the accessibility tree, including their backendNodeId. Their text content is hoisted as StaticText nodes under the nearest non-pruned ancestor, making sudoku-grid cells (and similar constructs) appear as plain, non-interactive text in the snapshot even though they are visually clickable. Fix: three-pronged approach in `find_cursor_interactive_elements` / `inject_cursor_orphans`: 1. JS parent-tagging: for every cursor-detected element, tag its parent with `data-__ab-ci-p=<idx>` so we can resolve parent backendNodeIds via a second `DOM.querySelectorAll` call. This avoids relying on `DOM.describeNode` returning `parentId`, which Chrome only emits for nodes already resolved in the CDP session. 2. inject_cursor_orphans (new function): for each cursor element whose backendNodeId is absent from the AX tree, create a synthetic `generic` TreeNode and attach it under the correct AX parent. 3. StaticText deduplication: after injecting a named generic, clear any matching StaticText sibling that Chrome hoisted from the pruned element, so the snapshot shows each cell exactly once rather than twice. Also adds 7 unit tests for `inject_cursor_orphans` covering: normal injection, empty-cell injection, StaticText clearing, non-matching StaticText preservation, skip-when-already-in-tree, skip-when-no-parent, and skip-when-parent-not-in-tree. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Additional Suggestion:
inject_cursor_orphans iterates the cursor_elements HashMap directly, so synthetic orphan cells are injected (and thus assigned refs / rendered) in non-deterministic order, making ref-to-cell mapping vary between identical snapshots.
…f order HashMap iteration is non-deterministic, so orphan cursor elements were being appended to the AX tree in a random order on each call. This caused ref numbers assigned to otherwise-identical snapshots to differ between runs (e.g. the first empty sudoku cell might be e4 one call and e11 the next). Sort `orphans` by `backend_node_id` before injection. Chrome assigns backendNodeIds incrementally as elements are created, so this also approximates DOM document order as a bonus. Addresses Vercel bot review comment on PR vercel-labs#1423. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Two issues prevented interacting with the Danske Spil Klub Lotto Sudoku game
(`danskespil.dk/klublotto/dagens-sudoku`):
Issue 1 — cross-origin iframe: `find_cursor_interactive_elements` was called
with the parent `session_id`, so JS ran against the main frame's DOM and missed
all elements inside the game's cross-origin iframe. Number-pad buttons had no refs.
Issue 2 — `role="none"` cells: Chrome completely omits elements with
`role="none/presentation"` from the AX tree (not even as ignored nodes). Their
text content is hoisted as `StaticText` under the nearest non-pruned ancestor.
All 81 sudoku grid cells appeared as plain, non-interactive `StaticText` items.
Before this fix:
```
```
After this fix (verified against live game with logged-in session):
```
... (all 81 grid cells have refs)
```
Fix
Commit 1 — iframe cursor detection (`effective_session_id`)
Switch `find_cursor_interactive_elements` from `session_id` to
`effective_session_id`. For cross-origin iframes, this is the iframe's own CDP
session (already computed by `resolve_ax_session`). For the main frame they are
identical — no behavioural change.
Commit 2 — grid cell detection (`inject_cursor_orphans`)
Three-pronged approach for elements with `role="none/presentation"` that Chrome
omits from the AX tree entirely:
JS parent-tagging: for every cursor-detected element, tag its
`parentElement` with `data-__ab-ci-p=`. A second
`DOM.querySelectorAll('[data-__ab-ci-p]')` then resolves parent `backendNodeId`s
without relying on `DOM.describeNode` returning `parentId` — which Chrome only
emits for nodes already resolved in the CDP session (they aren't, because
`DOM.getDocument(depth=0)` was used).
`inject_cursor_orphans`: for each cursor element whose `backendNodeId` is
absent from the AX tree, synthesise a `generic` `TreeNode` and attach it under
the correct AX parent found via the parent backendNodeId.
StaticText deduplication: Chrome hoists text content of pruned elements as
`StaticText` under the nearest non-pruned ancestor. After injecting the generic
cell, any matching `StaticText` sibling is cleared (`.clear()`) so each cell
appears exactly once — not twice.
Tests
7 new unit tests for `inject_cursor_orphans`:
All 760 tests pass.
Test plan
with `[ref]` and `clickable [cursor:pointer]`; number-pad buttons 1–9, SLET, HJÆLP
all have refs
🤖 Generated with Claude Code