Service Reliability Platform

A service reliability monitoring system built from scratch to simulate real-world microservices and operational monitoring.

Service events flow through Kafka and Spark to produce aggregated reliability metrics, which are stored in Delta tables and visualized in Grafana to track system health and deployment impact.

Architecture

┌──────────────┐    ┌─────────┐    ┌──────────────┐    ┌──────────────┐    ┌──────────────┐    ┌──────────────┐
│  Producers   │───▶│  Kafka  │───▶│    Bronze    │───▶│    Silver    │───▶│  Prometheus  │───▶│   Grafana    │
│  (Python)    │    │ (broker)│    │  (streaming) │    │  (streaming) │    │  (metrics)   │    │  (dashboards)│
│              │    │         │    │              │    │              │    │              │    │              │
│ api-gateway  │    │         │    │ Raw events   │    │ Windowed     │    │ :9090        │    │ :3000        │
│ auth         │    │         │    │ deduplicated │    │ aggregations │    │              │    │              │
│ payments     │    │         │    │              │    │              │    │              │    │              │
│ search       │    │         │    │              │    │ error_rate   │    │              │    │              │
│ deploy       │    │         │    │              │    │ p95_latency  │    │              │    │              │
└──────────────┘    └─────────┘    └──────────────┘    └──────────────┘    └──────────────┘    └──────────────┘
                                                              ↓
                                                     ┌──────────────┐
                                                     │     Gold     │
                                                     │   (batch)    │
                                                     │              │
                                                     │ Deploy       │
                                                     │ regression   │
                                                     │ detection    │
                                                     └──────────────┘

Tech Stack

Component	Technology
Event Generation	Python (Faker)
Message Broker	Apache Kafka (KRaft mode)
Stream Processing	Spark Structured Streaming
Storage	Delta Lake
Metrics Export	Python + prometheus_client
Monitoring	Prometheus
Visualization	Grafana
Orchestration	Docker Compose

Features

• Sliding Windows: Short sliding windows for near real-time aggregation
• Watermarking: Limited late-data tolerance to allow windows to close predictably
• Deduplication: By event_id to handle Kafka retries
• Deploy Regression Detection: Periodic batch analysis comparing before/after metrics
• Provisioned Dashboards: Grafana dashboards auto-load on startup

Quick Start

Prerequisites

• Docker Desktop
• Python 3.11+
• 8GB+ RAM recommended

1. Start Infrastructure

# Clone and navigate to project
cd service-reliability-platform

# Start all services
docker compose up -d

# Wait for Kafka to be ready (~30 seconds)
sleep 30

# Create Kafka topics
docker exec kafka kafka-topics --bootstrap-server kafka:29092 \
  --create --if-not-exists --topic service-events --partitions 3 --replication-factor 1

docker exec kafka kafka-topics --bootstrap-server kafka:29092 \
  --create --if-not-exists --topic deploy-events --partitions 1 --replication-factor 1

2. Start Producers

# Create virtual environment
python3 -m venv venv
source venv/bin/activate
pip install -r producers/shared/requirements.txt

# Run all producers
./scripts/run-producers.sh

3. Access Dashboards

Service	URL	Credentials
Grafana	http://localhost:3000	-
Prometheus	http://localhost:9090	-
Metrics Exporter	http://localhost:8000/metrics	-

4. Run Deploy Regression Analysis (Optional)

pip install pyspark==3.5.3 delta-spark==3.2.0
python scripts/run-gold-analysis.py

Metrics Exposed

Metric	Labels	Description
srp_request_count	service, region	Total requests in window
srp_error_count	service, region	Total errors in window
srp_error_rate	service, region	Error rate (0-1)
srp_avg_latency_ms	service, region	Average latency
srp_p95_latency_ms	service, region	P95 latency

Project Structure

├── docker-compose.yml          # All services orchestration
├── producers/                  # Python event generators
│   ├── api_gateway/
│   ├── auth/
│   ├── payments/
│   ├── search/
│   ├── deploy/
│   └── shared/                 # Common producer logic
├── streaming/                  # Spark streaming jobs
│   └── processors/
│       ├── bronze_ingestion.py # Raw event ingestion
│       └── silver_aggregation.py # Windowed aggregations
├── metrics_exporter/           # Prometheus exporter
│   └── exporter.py
├── monitoring/grafana/         # Grafana provisioning
│   ├── dashboards/
│   │   └── service-reliability.json
│   └── provisioning/
├── prometheus/                 # Prometheus config
├── scripts/                    # Utility scripts
│   ├── run-producers.sh
│   └── run-gold-analysis.py    # Batch deploy analysis
└── storage/                    # Delta Lake tables
    ├── bronze/
    ├── silver/
    └── gold/

Key Design Decisions

1. Bronze/Silver = Streaming, Gold = Batch: Bronze and Silver use streaming for continuous metrics, while Gold uses a batch job since deploy events are infrequent.

2. Custom Metrics Exporter: Prometheus scrapes processed data from Delta Lake, not raw Kafka events.

3. Provisioned Dashboards: No manual Grafana setup—dashboards load automatically via JSON.

4. Event Time Processing: Metrics are grouped based on event timestamps rather than ingestion time.

Stopping the Platform

# Stop all containers
docker compose down

# Clean all data (optional)
rm -rf storage/bronze storage/silver storage/gold storage/checkpoints