[Feature] Horizontal scaling — shared state backend for multi-instance deployments

[carlosatta]

What problem does this solve?

Routerly currently cannot run as multiple instances behind a load balancer. All state is either in-process memory or on-disk JSON files, both of which are node-local. Running two instances produces:

• Split routing state: routingMemoryStore (per-conversation routing history) and traceStore (request traces) are in-memory Maps — each node has a different view
• Split cache: semanticResponseCache is an in-memory Map — cache hits only occur if the same node handles the same project's requests
• Config write races: writeConfig() writes JSON files directly to disk — concurrent writes from two nodes corrupt the file
• Usage/budget split: usage records (usage.json) are written per-node, so budget enforcement reads incomplete data and limits can be exceeded

The result: horizontal scaling is currently impossible without data loss or incorrect behaviour.

Proposed solution

Extract all shared state behind a pluggable adapter interface, with at minimum a Redis/Valkey implementation alongside the existing in-process adapter (so single-node deployments keep working with zero config changes).

State that must move to shared storage:

Current location	What it stores	Suggested backend
routingMemoryStore.ts	Per-conversation routing history	Redis (TTL-keyed hash)
traceStore.ts	Request traces (5-min ring buffer)	Redis (TTL-keyed list)
semanticResponseCache.ts	Embedding → modelId cache	Redis (TTL-keyed hash)
config/*.json	Projects, models, users, settings	Redis or PostgreSQL
data/usage.json	Usage records for budget enforcement	PostgreSQL or Redis sorted set

Proposed config:

{
  "storage": {
    "adapter": "redis",
    "url": "redis://localhost:6379"
  }
}

Default adapter: "memory" preserves current single-node behaviour.

Config writes: replace file-based writeConfig() with atomic compare-and-swap on the shared backend to prevent concurrent-write corruption.

Alternatives you've considered

• Sticky sessions at the load balancer: routes the same project to the same node, partially addressing routing memory and cache misses. Does not solve config write races or usage record splits.
• Single-writer pattern: designate one node as config writer, others as read-only replicas. Reduces races but adds complexity and is still not truly stateless.

Who would benefit from this?

Any team running Routerly at scale where a single node is a bottleneck or single point of failure. Required for high-availability deployments.

Additional context

A clean separation between the state interface and its implementation would also make it easier to add alternative backends (e.g. PostgreSQL only, without Redis) in the future. The memory adapter can remain the default so existing Docker Compose deployments need no changes.