feat: route LLM calls through LiteLLM for multi-provider support

[prodmanpd]

Summary

Routes Cortex's LLM and embedding calls through the LiteLLM SDK instead of a hardcoded anthropic client and a hardcoded OpenAI embeddings path, so Cortex can run on any provider LiteLLM supports (Anthropic, OpenAI, Gemini, Bedrock, Vertex, Azure, Voyage, Cohere, or a LiteLLM proxy) via one interface.

• Backward compatible: Claude stays the default (claude-opus-4-5) for generation, and the embedding provider selection (model2vec, openai, hash) is unchanged. Existing setups keep working with no config change.
• Pick any generation provider by setting CORTEX_MODEL (for example openai/gpt-4o, gemini/gemini-2.5-pro, bedrock/...); pick any embedding provider by setting CORTEX_EMBEDDING_PROVIDER=litellm and a provider prefixed CORTEX_EMBEDDING_MODEL (for example voyage/voyage-3, cohere/embed-english-v3.0, gemini/text-embedding-004). Credentials come from each provider's own env var.
• drop_params=True on every call, so one config works across providers that reject each other's params.

Changes

• ui.py: ask_cortex() now calls litellm.completion(...); model from CORTEX_MODEL.
• ingest.py: extraction call now uses litellm.completion(...).
• backend/app/condense.py: condense call now uses litellm.completion(...)
• backend/app/extractor.py: _extract_with_claude now uses litellm.completion(...)

Embeddings

• backend/app/embeddings.py: adds litellm as a CORTEX_EMBEDDING_PROVIDER option backed by litellm.embedding (new _litellm_embedding). model2vec, openai, and hash paths are untouched. Keeps the existing safety net: a dimension mismatch raises so retrieval degrades to the deterministic hash embedding (or fails in strict mode) rather than corrupting the vector index. Optional CORTEX_LITELLM_BASE_URL and CORTEX_LITELLM_API_KEY point at a LiteLLM proxy.

Packaging

• requirements.txt, backend/requirements.txt: add litellm>=1.89.0,<2.0.0

Tests

1. Compile: python -m py_compile ui.py ingest.py backend/app/condense.py backend/app/extractor.py backend/app/embeddings.py gives OK.

2. Live call through a converted chat path (the exact litellm.completion shape the four sites use, a system prompt plus a single user turn, drop_params=True), against a live model via a LiteLLM proxy:

cortex pattern -> 'OK'
usage: {'prompt_tokens': 26, 'completion_tokens': 1, 'total_tokens': 27}

3. Embedding provider route (litellm stubbed, no network):

1) voyage route -> litellm dim 8 | drop_params True
2) text-embedding-3 route -> dimensions forwarded: 8
3) mismatch (5 vs 8) non-strict -> hash (safe fallback)
4) status: {'configured_provider': 'litellm', 'network_required': True}
ALL EMBEDDING TESTS PASSED

This confirms CORTEX_EMBEDDING_PROVIDER=litellm routes through litellm.embedding, forwards an output dimension only for text-embedding-3 models, and degrades to hash on a dimension mismatch so the index stays consistent.

4. Pin resolves cleanly: pip install "litellm>=1.89.0,<2.0.0"

Risk and compatibility

• Additive and backward compatible: default generation model unchanged; default embedding provider selection unchanged; existing ANTHROPIC_API_KEY and OPENAI_API_KEY setups keep working.
• anthropic stays a dependency; the app no longer instantiates the Anthropic client directly.
• No DB, schema, or API changes.

Example usage

# Default: Claude for generation, existing embedding provider, as before:
export ANTHROPIC_API_KEY=sk-ant-...

# Any generation provider, no code change:
export CORTEX_MODEL="openai/gpt-4o"      && export OPENAI_API_KEY=sk-...
export CORTEX_MODEL="gemini/gemini-2.5-pro" && export GEMINI_API_KEY=...

# Any embedding provider via LiteLLM:
export CORTEX_EMBEDDING_PROVIDER=litellm
export CORTEX_EMBEDDING_MODEL="voyage/voyage-3" && export VOYAGE_API_KEY=...

[liuzhao1225]

I'm concerned about the supply-chain boundary of this LiteLLM integration. The upstream incident report confirms compromised PyPI releases 1.82.7/1.82.8 in March 2026; those releases were removed.

At reviewed head c3bc8c4e7a, this PR includes litellm>=1.89.0,<2.0.0 in backend/requirements.txt; litellm>=1.89.0,<2.0.0 in requirements.txt.

That upper bound permits later 1.x releases beyond a single tested version when dependencies are resolved afresh or upgraded. Please document tested versions, artifact verification, and whether installation of the SDK is opt-in. A disabled application feature cannot prevent an installed malicious Python-startup hook.

This is one of 49 observed LiteLLM integration PRs from the same account, whose author acknowledged the cross-project effort. The pattern prompted this review; it does not establish malicious intent or connect the author to the incident. The consolidated questions and corrections are in YouDub #130; this PR's review snapshot preserves the revision and scope.