Skip to content

fix(enrichment): circuit-break exhausted quota calls - #229

Open
mikemikimike wants to merge 5 commits into
verygoodplugins:developfrom
mikemikimike:fix/222-enrichment-quota-circuit
Open

fix(enrichment): circuit-break exhausted quota calls#229
mikemikimike wants to merge 5 commits into
verygoodplugins:developfrom
mikemikimike:fix/222-enrichment-quota-circuit

Conversation

@mikemikimike

Copy link
Copy Markdown

Summary

Closes #222.

When the enrichment LLM returns a definitive quota exhaustion error, repeated stores previously retried summarization and classification independently. This adds a shared, fail-soft cooldown circuit across both enrichment paths.

Changes

  • Open the circuit on insufficient_quota/quota errors for a configurable ENRICHMENT_CIRCUIT_COOLDOWN_SECONDS period (default 300 seconds).
  • Skip additional upstream LLM requests while the circuit is open, while preserving deterministic fallback behavior.
  • Allow one probe after cooldown and close the circuit after a successful response.
  • Expose circuit-open skips and recoveries through the enrichment status response.
  • Add focused regression tests for open, skip, non-quota, and recovery behavior.

The default remains backward compatible for successful requests and non-quota transient failures. The change only suppresses further enrichment requests after a quota-related failure; stores remain fail-soft.

Verification

  • python -m compileall -q automem app.py — passed.
  • Direct circuit regression assertions — passed.
  • python -m black --check app.py automem tests/test_enrichment_circuit.py — passed.
  • python -m flake8 app.py automem --count --select=E9,F63,F7,F82 --show-source --statistics — passed.
  • python -m pytest -q tests/test_enrichment_circuit.py — not runnable in this environment: pytest collection resolves an unrelated installed tests package (D:\project\tscodex\connectonion-pr\tests) and cannot import this checkout's tests.support; no test assertion was executed.
  • Full make test and Docker integration tests were not run because the focused pytest collection issue must be resolved before they can provide meaningful results.

@mikemikimike

Copy link
Copy Markdown
Author

Follow-up validation: reproduced and fixed a probe-state bug where a non-quota failure after cooldown could leave the circuit permanently blocking requests. Added regression coverage for probe recovery, classifier request suppression, summarizer request suppression, and non-quota behavior. Verified with compileall, Black, focused Flake8, and direct classifier/circuit assertions. Full pytest collection remains blocked locally by the unrelated installed tests package collision documented in the PR.

@mikemikimike

Copy link
Copy Markdown
Author

Docker verification completed: docker compose run --rm flask-api sh -c "pip install --no-cache-dir -r requirements-dev.txt && python -m pytest -q tests --ignore=tests/benchmarks" — 664 passed, 12 skipped, 3 warnings in 6:06:41. The warnings are deprecations from huggingface_hub/hf_xet; no test failures.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(enrichment): circuit-break exhausted-quota LLM calls

2 participants