fix(gateway): mid-tier failover for 402 spending-limit

[yunfanxing6]

Summary

Makes Grok2API behave like a mid-tier relay for Grok Build 402 personal-team-blocked:spending-limit failures:

1. Always rotate accounts on 402 — ignore upstream X-Should-Retry: false for payment-required so multi-account failover actually runs.
2. Pause exhausted accounts — free / unrecognized billing profiles are removed from the hot pool for freeQuotaRecoveryPause (2h, was hardcoded 24h for free recovery). Paid profiles with a known period end still wait until period end. Failed probes re-apply the pause.
3. Never surface upgrade/credit prompts to clients — map 402 / quota exhaustion to a sanitized 503 upstream_unavailable so Grok CLI / OpenAI-compatible clients do not pop "Add credits / SuperGrok" dialogs.

Motivation

In production we observed:

• Upstream returns 402 + body personal-team-blocked:spending-limit / "You have run out of credits..."
• Adapter/gateway treated X-Should-Retry: false as final → attempt_count=1 and raw 402 body passthrough
• MarkPaidQuotaExhausted returned false when billing plan fields were empty → accounts only got short cooldown and re-entered the pool
• Grok CLI interpreted the passthrough body as a billing wall (upgrade / buy credits UI)

Behavior after this change

Event	Behavior
Single account 402	Exclude account, try next (up to routing.maxAttempts)
Free/unknown billing 402	quota_recovery free exhausted, next_probe_at = now+2h
Paid with period end	quota_recovery paid exhausted until period end
All attempts fail	Client gets 503 sanitized error, not official upgrade text
Success on later account	Normal 200 stream/JSON

Test plan

• Unit: TestPaymentRequiredAlwaysRetriesDespiteUpstreamVeto — 402 + X-Should-Retry: false remains retryable
• Production smoke (self-hosted multi-account pool):
  • Normal chat → 200
  • Same request_id logs multiple upstream_request_failed with different account_id on 402
  • Client HTTP status becomes 503 after pool failure (not 402)
  • Exhausted accounts appear in account_quota_recovery with ~2h next_probe_at
  • Response body does not contain "Add credits" / "supergrok"

go test ./backend/internal/application/gateway/ -count=1

Notes / non-goals

• Does not invent quota that does not exist: if every account is exhausted, clients still get a temporary error (503), which is preferable to a fake upgrade wall.
• freeQuotaRecoveryPause = 2h is a product choice for mid-tier relay freshness; happy to make it a routing / runtime setting in a follow-up if maintainers prefer.