🧠 LLM Inference Logging & Ingestion Platform

A production-grade, full-stack LLM observability platform with multi-provider chatbot, real-time inference logging, event-driven ingestion, and monitoring dashboards — all runnable with a single command.

✨ Features

• Multi-turn Chatbot UI — Streaming chat with OpenAI, Anthropic, Gemini, and OpenRouter
• Lightweight SDK — Async LLMLogger wrapper capturing latency, TTFB, tokens, and errors
• Event-Driven Ingestion — Kafka (Redpanda) pipeline with dead letter queue and retry logic
• PII Redaction — Microsoft Presidio anonymizes sensitive data before storage
• Observability Dashboards — Pre-built Grafana panels for latency, throughput, errors
• Conversation Management — List, cancel, and resume conversations from the UI
• Docker Compose — One command brings up all 8 services

📸 Screenshots

1. Chat Interface (Markdown & Tables)

2. Live Observability (Grafana)

3. Conversation Management

---

🚀 Full Setup & Testing Guide

This project is completely containerized. You do not need to install Node or Python on your host machine—only Docker.

1. Prerequisites

• Docker & Docker Compose v2+ installed on your machine.
• An API key for at least one provider (OpenAI, Anthropic, Google, or OpenRouter).

2. Configure API Keys

Clone the repository and set up your environment variables:

git clone https://github.com/your-username/llm-inference-platform
cd llm-inference-platform
cp .env.example .env

Open the .env file in your favorite editor and paste your API keys. It should look like this:

# Provide at least one of these:
OPENAI_API_KEY=sk-proj-...
ANTHROPIC_API_KEY=sk-ant-...
GOOGLE_API_KEY=AIza...
OPENROUTER_API_KEY=sk-or-v1-...

# Optional: Disable PII redaction to speed up the system slightly
PII_REDACTION_ENABLED=true

3. Spin Up the Infrastructure

Run the following command to build and start all 8 distributed services:

docker compose up --build -d

Note: The very first build will take about ~3-5 minutes as it downloads the Python images, Next.js dependencies, and the ~500MB SpaCy NLP model used for PII redaction. Subsequent boots will be instant.

4. Verify Services are Running

Check that all containers are healthy:

docker compose ps

You should see frontend, chat-api, ingestion-worker, postgres, redpanda, prometheus, and grafana in the Up or Healthy state.

5. How to Test the Application

1. Test the Chatbot (Frontend):

   • Open http://localhost:3000 in your browser.
   • Select a provider from the top dropdown (e.g., OpenRouter or OpenAI).
   • Type a prompt like: "Write a Python script to reverse a string."
   • Notice how the response streams in real-time, just like ChatGPT.
   • Look at the sidebar: Your chat title dynamically updates to match your prompt!

2. Test Conversation Management:

   • Click "New Chat" and start a second conversation.
   • Click back to your first conversation in the sidebar. Notice how it instantly resumes with full context fetched from the database.

3. Test the Observability Pipeline (Grafana):

   • Open http://localhost:3001 in a new tab.
   • Login with admin / admin.
   • Open the LLM Platform Main Dashboard.
   • You will see the total requests, P95 latency, and token throughput for the messages you just sent! (The backend SDK asynchronously pushed these to Kafka, which were ingested and exported to Prometheus).

4. Test the API directly (Optional):

   • View the interactive Swagger docs at http://localhost:8000/docs.
   • You can programmatically stream a response via curl:

     curl -N -X POST http://localhost:8000/chat/stream \
       -H "Content-Type: application/json" \
       -d '{"message": "Hello!", "provider": "openai", "model": "gpt-4o"}'

---

🏗️ Architecture

📖 Note for Reviewers: For a comprehensive deep-dive into the ingestion lifecycle, scaling strategies, bottlenecks, and failure handling assumptions, please read the full ARCHITECTURE.md document.

┌─────────────────────────────────────────────────────────────┐
│                   Frontend (Next.js 14)                      │
│         Chat UI · Conversations · Grafana Dashboard          │
└───────────────────────┬─────────────────────────────────────┘
                        │ HTTP / SSE Streaming
┌───────────────────────▼─────────────────────────────────────┐
│                  Chat API (FastAPI :8000)                    │
│  LLM Providers · LLMLogger SDK · PII Redaction · Metrics    │
└──────────┬────────────────────────────┬─────────────────────┘
           │ Kafka Produce              │ Async DB Writes
           ▼                           ▼
┌──────────────────┐     ┌─────────────────────────────────────┐
│  Redpanda/Kafka  │     │         PostgreSQL 16                │
│  topic: llm-logs │     │  sessions · messages · logs · dlq   │
└──────────┬───────┘     └─────────────────────────────────────┘
           │ Consume                   ▲
┌──────────▼──────────────────────────┴───────────────────────┐
│               Ingestion Worker (Python)                      │
│   Kafka Consumer · Pydantic Validation · Retry · DLQ        │
└─────────────────────────────────────────────────────────────┘
           │ Prometheus /metrics
┌──────────▼──────────────────────────────────────────────────┐
│            Prometheus + Grafana                              │
│  Latency P50/P95/P99 · Error Rate · Throughput · Tokens     │
└─────────────────────────────────────────────────────────────┘

Ingestion Flow

1. User sends a message via the chat UI
2. Chat API calls the selected LLM provider (streaming SSE)
3. LLMLogger SDK wraps the call, capturing: request_ts, ttfb_ms, total_latency_ms, input_tokens, output_tokens, status
4. PII Redactor (Presidio) strips emails, phones, SSNs from previews
5. SDK fire-and-forgets a Kafka produce to topic llm-logs — never blocking the stream
6. Ingestion Worker consumes the event, validates with Pydantic, upserts into PostgreSQL
7. On validation failure → Dead Letter Queue (dead_letter_logs table)
8. Prometheus metrics emitted; Grafana panels auto-update

---

📊 Database Schema

Design Decisions

Four tables with intentional tradeoffs:

Table	Purpose	Key Decision
sessions	Conversation lifecycle	status field enables soft cancel/resume
messages	Full chat history	Dual content/content_redacted fields for PII-safe analytics
inference_logs	One row per LLM API call	raw_metadata JSONB for forward-compat with new provider fields
dead_letter_logs	Failed ingestion events	Kafka coordinates stored for manual replay

Why PostgreSQL over ClickHouse?

• Simpler ops for MVP with a single Docker container
• JSONB handles flexible provider metadata without migrations
• At scale (>1M logs/day), migrate inference_logs to ClickHouse for time-series queries

Why content_redacted separate from content?

• Analytics and debugging can use the PII-free version safely
• Full content retained for compliance audits (toggle with STORE_RAW_CONTENT=false)

Schema

-- sessions: one row per conversation
sessions (id UUID PK, status TEXT, provider TEXT, model TEXT, message_count INT, title TEXT, metadata JSONB, created_at, updated_at)

-- messages: ordered chat turns
messages (id UUID PK, session_id FK, role TEXT, content TEXT, content_redacted TEXT, sequence_num INT, token_count INT, created_at)

-- inference_logs: one row per LLM API call
inference_logs (id UUID PK, session_id FK, message_id FK, provider TEXT, model TEXT, request_ts, response_ts, total_latency_ms INT, ttfb_ms INT, input_tokens INT, output_tokens INT, total_tokens INT, status TEXT, error_type TEXT, http_status INT, input_preview TEXT, output_preview TEXT, raw_metadata JSONB, created_at)

-- dead_letter_logs: failed events for replay
dead_letter_logs (id UUID PK, raw_payload JSONB, error_reason TEXT, created_at)

---

🔧 Configuration

All settings via environment variables (see .env.example):

Variable	Default	Description
OPENAI_API_KEY	—	OpenAI API key
ANTHROPIC_API_KEY	—	Anthropic API key
GOOGLE_API_KEY	—	Google Gemini API key
DEFAULT_PROVIDER	openai	Default LLM provider
DEFAULT_MODEL	gpt-4.1	Default model
DATABASE_URL	postgres://...	Async PostgreSQL URL
KAFKA_BOOTSTRAP_SERVERS	redpanda:9092	Kafka broker
KAFKA_TOPIC	llm-logs	Inference log topic
PII_REDACTION_ENABLED	true	Enable Presidio redaction
STORE_RAW_CONTENT	true	Store unredacted content
OPENROUTER_API_KEY	—	OpenRouter API key

---

🎭 Multi-Provider Support

The LLMProvider abstract base class normalizes the interface across providers:

provider = get_provider("openai")   # or "anthropic" / "google" / "openrouter"
async for token in provider.stream(messages, model="gpt-4.1"):
    yield token

Supported models:

Provider	Models
OpenAI	gpt-4.1, gpt-4o, gpt-3.5-turbo
Anthropic	claude-sonnet-4-5, claude-haiku-3-5
Google	gemini-1.5-pro, gemini-1.5-flash
OpenRouter	llama-3-8b-instruct, gemini-flash-1.5, claude-3-haiku, gpt-4o-mini

---

🔐 PII Redaction

Using Microsoft Presidio:

• Entities detected: EMAIL, PHONE, CREDIT_CARD, SSN, PERSON, IP_ADDRESS, LOCATION, DATE
• Applied to input_preview and output_preview in inference logs (first 200 chars)
• Applied to content_redacted in messages table
• Configurable: set PII_REDACTION_ENABLED=false to disable

---

📈 Logging Strategy

The LLMLogger SDK is designed for zero-overhead observability:

1. Async fire-and-forget: Kafka produce never blocks the response path
2. Lazy initialization: Kafka producer created once via class-level singleton
3. Graceful fallback: If Kafka is unavailable, falls back to stderr logging
4. TTFB tracking: First token latency captured separately from total latency
5. Usage sentinel: Providers emit a hidden __usage__:N:M token to pass token counts through the stream without breaking SSE

---

⚡ Scaling Considerations

Concern	Strategy
High message volume	Kafka partitioning by session_id; scale ingestion consumers horizontally (kafka_group_id consumer group)
DB write pressure	Batched upserts in ingestion worker; connection pooling (pool_size=10)
Large context windows	Chat API trims history to last N messages (configurable)
PII redaction overhead	Only applied to 200-char previews, not full content
SDK latency impact	<1ms added; async produce never awaited in request path
Cold start	Redpanda + Postgres healthchecks ensure services are ready before dependents start

---

🔴 Failure Handling

Failure	Behavior
Kafka unavailable (produce)	SDK catches exception, logs to stderr; chat response unaffected
Kafka unavailable (consume)	Worker retries connection with exponential backoff
Invalid payload	Sent to dead_letter_logs; offset committed; worker continues
DB write fails (transient)	3 retries with exponential backoff (1s, 2s, 4s)
DB write fails (permanent)	Sent to dead letter; offset committed
LLM provider error	Error captured in inference_logs.status='error'; SSE sends error event
Client disconnect	asyncio.CancelledError caught; partial message saved as cancelled

---

🐳 Docker Services

Service	Image	Port	Health Check
frontend	node:20-alpine	3000	HTTP /api/health
chat-api	python:3.12-slim	8000	HTTP /health
ingestion-worker	python:3.12-slim	8001	HTTP /health
postgres	postgres:16-alpine	5432	pg_isready
redpanda	redpandadata/redpanda	9092, 9644	rpk cluster health
prometheus	prom/prometheus	9090	—
grafana	grafana/grafana	3001	—

---

☸️ Kubernetes (Bonus)

Kubernetes manifests are in k8s/:

kubectl apply -f k8s/namespace.yaml
kubectl apply -f k8s/

See k8s/README.md for minikube setup instructions.

---

🛠️ What I'd Improve With More Time

1. ClickHouse for inference_logs time-series queries at scale (10x faster aggregations)
2. OpenTelemetry for end-to-end distributed traces across all services
3. Alembic schema migrations instead of raw init.sql
4. Auth (JWT/OAuth) for multi-tenant conversation isolation
5. Grafana Alerting — Slack/PagerDuty on error rate spikes or P99 > 10s
6. Dead Letter Replay API — HTTP endpoint to re-process failed events from DLQ
7. Model cost tracking — Input/output token pricing per provider/model
8. Rate limiting — Per session/API key request throttling
9. Streaming token buffering — Batch small tokens before SSE flush for smoother UX
10. Conversation search — Full-text search over message history

---

📁 Project Structure

llm-inference-platform/
├── docker-compose.yml          # All 8 services
├── .env.example                # Environment variable template
├── README.md
│
├── chat-api/                   # FastAPI backend
│   ├── main.py                 # App entry + lifespan
│   ├── config.py               # Pydantic settings
│   ├── metrics.py              # Prometheus counters/histograms
│   ├── providers/              # LLM provider implementations
│   │   ├── base.py             # Abstract LLMProvider
│   │   ├── openai_provider.py
│   │   ├── anthropic_provider.py
│   │   ├── gemini_provider.py
│   │   └── openrouter_provider.py
│   ├── sdk/                    # LLMLogger SDK
│   │   ├── logger.py           # Core instrumentation
│   │   └── pii_redactor.py     # Presidio integration
│   ├── routers/
│   │   ├── chat.py             # SSE streaming endpoint
│   │   ├── conversations.py    # CRUD + cancel/resume
│   │   └── health.py           # Health + Prometheus metrics
│   └── db/
│       ├── models.py           # SQLAlchemy ORM models
│       └── session.py          # Async session factory
│
├── ingestion-worker/           # Kafka consumer
│   ├── main.py                 # Entry point + HTTP server
│   ├── consumer.py             # Kafka consume loop
│   ├── processor.py            # DB upserts + metrics
│   ├── schemas.py              # Pydantic validation
│   ├── metrics.py              # Ingestion-specific metrics
│   └── db/                     # Shared ORM models
│
├── frontend/                   # Next.js 14
│   ├── app/
│   │   ├── page.tsx            # Chat UI
│   │   ├── conversations/      # Conversation list
│   │   └── dashboard/          # Grafana embed
│   ├── components/
│   │   ├── MessageBubble.tsx
│   │   ├── Sidebar.tsx
│   │   ├── ProviderSelector.tsx
│   │   └── TypingIndicator.tsx
│   └── lib/api.ts              # API client
│
├── infra/
│   ├── init.sql                # PostgreSQL schema
│   ├── prometheus.yml          # Scrape config
│   └── grafana/                # Pre-built dashboards
│
└── k8s/                        # Kubernetes manifests
    ├── namespace.yaml
    ├── chat-api-deployment.yaml
    ├── ingestion-deployment.yaml
    ├── postgres-statefulset.yaml
    └── kafka-statefulset.yaml

---