Skip to content

fix(playground): classify Gemini errors from their message, not metadata - #133

Merged
leoafarias merged 1 commit into
mainfrom
fix/error-classifier-request-id
Sep 23, 2026
Merged

leoafarias merged 1 commit into
mainfrom
fix/error-classifier-request-id

Conversation

@leoafarias

Copy link
Copy Markdown
Collaborator

Intent

Fix a real misclassification of Gemini errors in the Wizard. It was also the cause of the intermittent generation_model_client_test failures in CI (#129 and #132 each failed once on it today).

Root cause

  • googleai_dart stamps every request with generateRequestId() = 'req_${DateTime.now().millisecondsSinceEpoch}'. ApiException.toString() appends that ID, the request URL, and the latency (403ms). This is unchanged in 13.0.0.
  • ErrorClassifier checks 401/403/408/504/429 status codes structurally. For everything else, it substring-matched the whole toString() against patterns that include bare numbers: '429' for rate limits, and '401'/'403' for auth.
  • Whenever the millisecond timestamp contained one of those numbers, the error was misclassified. Two examples:
    • A 400 "blocked by safety filters" was reported to the user as "API key is invalid".
    • A 503 "service unavailable" was reported as "The model is overloaded".

Fix (error_classifier.dart)

  • googleai_dart errors (the sealed GoogleAIException) are matched on their message, not toString(), so the URL, request ID and latency never reach the patterns.
  • Status-number patterns match only as whole numbers (\b429\b). A standalone "Error 429:" or "(403)" still matches; digits inside an ID, a timestamp or a duration don't. Word patterns keep substring matching, so their behavior is unchanged.

Tests

New deterministic regressions build an ApiException with request ID req_1790401403429 and a 403 ms latency. They check that a 400 safety block stays a safety block, a 503 "unavailable" stays unknown, and "API key not valid" is still recognized as authentication from the message. A plain-string case (14290 ms, job 1403) covers whole-number matching.

  • All 4 regressions failed before the fix, each misclassified as a rate limit, and pass after it.
  • The existing classifier tests pass unchanged.

Impacted packages

playground.

Commands run

  • fvm dart analyze --fatal-infos in playground, including test_live: no issues.
  • fvm flutter test in playground: 375 passing.
  • generation_model_client_test.dart run 30 times in a row: 30/30 passing.

googleai_dart appends the request URL, a millisecond-timestamp request ID (req_1790…), and the latency to an exception's toString(). ErrorClassifier substring-matched that whole string against status-number patterns, so whenever the timestamp contained 401, 403, or 429 a safety block or a server error was reported as an invalid API key or an overloaded model. That also made generation_model_client_test fail intermittently in CI.

Match SDK errors on their message, and match status numbers only as whole numbers.
@github-actions

Copy link
Copy Markdown

Visit the preview URL for this PR (updated for commit 8e04574):

https://superdeck-dev--pr133-fix-error-classifier-f3vnjquw.web.app

(expires Fri, 23 Oct 2026 04:56:20 GMT)

🔥 via Firebase Hosting GitHub Action 🌎

Sign: bd68fc230762285849207e7e120aaf87cd4ca2f9

@leoafarias
leoafarias merged commit beb71a6 into main Sep 23, 2026
5 checks passed
@leoafarias
leoafarias deleted the fix/error-classifier-request-id branch September 23, 2026 05:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant