RouterQ

Secure and Efficient Local and Private LLM router written in Rust.

Features

• Optimized routing — score providers by cost, latency, capability, or balanced; round-robin; automatic failover
• Token compression — local string transforms and/or private LLM semantic compression (light / medium / aggressive)
• Guardrails — detect & redact (or block) email, phone, SSN, credit cards (Luhn), API keys, JWTs, IPs, custom regex
• Terminal TUI — chat, provider health, live stats, event log
• Web UI — browser dashboard to manage providers, settings, chat playground, and live stats

Quick start

# Generate config
cargo run --release -- --init-config

# Offline demo (mock provider, no API keys)
cargo run --release -- --prompt "Hello from RouterQ"

# Interactive TUI
cargo run --release

# Web management UI (http://127.0.0.1:8787)
cargo run --release -- --web

API keys are not read from env or stored in TOML. Set them at runtime after start (Web UI → Providers → Edit, or the REST endpoints below). Keys live only in process memory.

Web UI

cargo run --release -- --web
# custom bind address
cargo run --release -- --web --listen 0.0.0.0:9000

Open the URL printed in the terminal. The dashboard covers:

Section	What you can do
Dashboard	Live stats, strategy, compression / guardrail / failover toggles
Chat	Send prompts through the full pipeline
Providers	Health, metrics, enable / disable
Settings	Router, compression, and guardrail options
Events	Live event log

Changes apply at runtime. Save config writes the current settings back to config.toml.

REST API

Method	Path	Description
GET	/api/overview	Stats + feature flags
GET	/api/providers	Provider list + metrics
POST	/api/providers	Create provider (optional api_key runtime-only)
PUT	/api/providers/:id	Update provider (optional api_key runtime-only)
PUT	/api/providers/:id/enabled	{ "enabled": true/false }
PUT	/api/providers/:id/api-key	{ "api_key": "sk-..." } (memory only)
DELETE	/api/providers/:id	Remove provider
GET	/api/config	Full config snapshot (no secrets)
PUT	/api/config/router	Update router section
PUT	/api/config/compress	Update compression
PUT	/api/config/compress/api-key	{ "api_key": "..." } compress LLM key
PUT	/api/config/guard	Update guardrails
PUT	/api/config/guard/api-key	{ "api_key": "..." } guard LLM key
POST	/api/config/save	Persist config to disk (never writes API keys)
PUT	/api/strategy	{ "strategy": "balanced" }
POST	/api/toggle/compress	Toggle compression
POST	/api/toggle/guard	Toggle guardrails
POST	/api/chat	Standard { "messages": [...] } or legacy { "prompt": "..." } → RouterQ response
POST	/v1/chat/completions	OpenAI-compatible chat completions (also /chat/completions)
GET	/api/events	Event log
GET	/api/stats	Aggregate stats

Chat API (standard LLM format)

RouterQ accepts the common OpenAI chat-completions body:

{
  "model": "routerq",
  "messages": [
    { "role": "system", "content": "You are a helpful assistant." },
    { "role": "user", "content": "Hello" }
  ],
  "temperature": 0.7,
  "max_tokens": 1024
}

• POST /v1/chat/completions — OpenAI-compatible response (choices, usage, …) plus optional routerq metadata (provider, cost, redactions).
• POST /api/chat — same request body; returns RouterQ’s native RouteResponse (or still accepts legacy { "prompt": "..." }).
• model is accepted for client compatibility; the routed provider’s model is used.
• stream: true is rejected (non-streaming only for now).

Example with the official OpenAI Python SDK:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8787/v1", api_key="not-needed")
resp = client.chat.completions.create(
    model="routerq",
    messages=[{"role": "user", "content": "Hello from RouterQ"}],
)
print(resp.choices[0].message.content)

TUI keys

Key	Action
Enter	Send prompt
Tab	Next view
s	Cycle routing strategy
c	Toggle compression
g	Toggle guardrails
q / Esc	Quit

Pipeline

input → guardrails (regex | private LLM | hybrid) → compress (local | private LLM | hybrid) → rank providers → complete (+ failover) → stats

Private / small LLM redactor

Use a local or private OpenAI-compatible model to filter sensitive data before the prompt is sent to the main provider:

[guard]
engine = "hybrid"   # "regex" | "llm" | "hybrid"

[guard.llm]
base_url = "http://127.0.0.1:11434/v1"   # Ollama OpenAI-compatible API
model = "llama3.2:1b"
timeout_secs = 15
fallback_to_regex = true
# API key: set at runtime (Web UI Settings or PUT /api/config/guard/api-key)

Engine	Behavior
regex	Fast pattern matching only (default)
llm	Call the private model for semantic redaction
hybrid	Regex first, then private LLM on the residual text

Private / small LLM compressor

Use the same class of private model to shrink prompts while preserving intent:

[compress]
engine = "hybrid"   # "local" | "llm" | "hybrid"
level = "medium"

[compress.llm]
base_url = "http://127.0.0.1:11434/v1"
model = "llama3.2:1b"
timeout_secs = 15
fallback_to_local = true
# API key: set at runtime (Web UI Settings or PUT /api/config/compress/api-key)

Engine	Behavior
local	Fast string transforms only (default)
llm	Call the private model for semantic compression
hybrid	Local first, then private LLM on the residual text

Example with Ollama:

ollama pull llama3.2:1b
# ensure Ollama is serving OpenAI-compatible /v1

Then set engine = "llm" or "hybrid" and restart / apply settings in the Web UI.

Config

See config.example.toml. API keys are never stored in config or environment variables; set them on the fly via Web UI or REST after the process starts.