fix(web-search): accept div result titles in Brave keyless scraper#719
Closed
tongshu2023 wants to merge 1 commit into
Closed
fix(web-search): accept div result titles in Brave keyless scraper#719tongshu2023 wants to merge 1 commit into
tongshu2023 wants to merge 1 commit into
Conversation
…HU-MAIC#687) Brave moved the result title from a span with class search-snippet-title to a div (class "title search-snippet-title line-clamp-1"), so the title regex matched nothing and every snippet was skipped - the keyless path returned 0 results on a 200 response with 20 snippets. Accept either element via a backreferenced (span|div) group. Test fixtures updated to the current div markup, with one legacy span case kept for back-compat. Verified live: search.brave.com currently serves 20/20 div titles and 0 span titles. Fixes THU-MAIC#687 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
Author
|
Closing as a duplicate of #688, which predates this PR and fixes the same root cause (I missed it before opening — apologies for the noise). Leaving my live-verification data over there in support. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #687. The keyless Brave web-search path returned 0 results because Brave moved the result title element:
<span class="search-snippet-title">Title</span><div class="title search-snippet-title line-clamp-1 ..." title="Title">Title</div>The title regex matched
<span>only, soif (!title) continue;skipped every snippet.Changes
parseBraveSearchHtmlaccepts<span>or<div>for the title via a backreferenced(span|div)group, so the closing tag must match the opening one.<div>markup; one legacy<span>case kept for back-compat (per the issue's suggestion).Verification
https://search.brave.com/search?q=photosynthesisreturns HTTP 200 with 20data-type="web"snippets — 0 span titles, 20 div titles. The old parser extracts 0 of them; the fixed one extracts them all.vitest run tests/web-search/brave.test.ts— 3/3 pass;prettier --checkandeslintclean.🤖 Generated with Claude Code