Skip to content

Fix Corrective-RAG never triggering web search (exact-match grade parsing) #246

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
douxiao398 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Fix Corrective-RAG never triggering web search (exact-match grade parsing) #246

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions corrective-rag/workflow.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,13 +127,21 @@ async def eval_relevance(
relevancy = self.llm.complete(prompt)
relevancy_results.append(relevancy.text.lower().strip())

# The grader is asked for a binary 'yes'/'no', but LLMs often add
# punctuation or a short justification (e.g. "No, ..."). Match on the
# leading token so the decision is robust instead of requiring the result
# to be exactly "yes"/"no".
relevant_texts = [
retrieved_nodes[i].text
for i, result in enumerate(relevancy_results)
if result == "yes"
if result.startswith("yes")
]
relevant_text = "\n".join(relevant_texts)
if "no" in relevancy_results:
# Trigger a corrective web search whenever at least one retrieved document
# was judged not relevant (anything that isn't a clear "yes"). The previous
# `"no" in relevancy_results` only matched a result that was exactly "no",
# so answers like "No, ..." or "no." silently skipped the web search.
if any(not result.startswith("yes") for result in relevancy_results):
return WebSearchEvent(relevant_text=relevant_text)
else:
return QueryEvent(relevant_text=relevant_text, search_text="")
Comment on lines +144 to 147

@coderabbitai coderabbitai Bot Jun 18, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Handle empty retrieval as a web-search trigger.

At Line 144, any(...) is false for an empty relevancy_results, so zero retrieved docs bypass corrective search and go straight to QueryEvent. That misses the fallback path exactly when retrieval fails.

Suggested fix 
-        if any(not result.startswith("yes") for result in relevancy_results):
+        if not relevancy_results or any(
+            not result.startswith("yes") for result in relevancy_results
+        ):
             return WebSearchEvent(relevant_text=relevant_text)
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@corrective-rag/workflow.py` around lines 144 - 147, The condition at line 144
using `any(not result.startswith("yes") for result in relevancy_results)`
evaluates to False when relevancy_results is empty, causing the code to
incorrectly return QueryEvent instead of triggering a web search. Modify the
condition to explicitly check if relevancy_results is empty OR if any result
doesn't start with "yes", so that empty retrieval results trigger the
WebSearchEvent fallback path as intended for corrective search behavior.

Expand Down
7 changes: 6 additions & 1 deletion firecrawl-agent/workflow.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -180,7 +180,12 @@ async def eval_relevance(
print(f"DEBUG: Relevant texts count: {len(relevant_texts)}")
print(f"DEBUG: Relevant text preview: {relevant_text[:200]}...")

if "no" in relevancy_results_striped:
# A document is treated as relevant when its grade contains "yes" (see the
# filter above), so trigger a corrective web search whenever any document is
# *not* relevant. The previous `"no" in relevancy_results_striped` only
# matched a grade equal to exactly "no", so answers like "No, ..." or "no."
# silently skipped the web search.
if any("yes" not in result.lower() for result in relevancy_results_striped):
print("DEBUG: Some documents irrelevant, returning WebSearchEvent")
return WebSearchEvent(relevant_text=relevant_text)
else:
Expand Down