inference-lab

Deploying LLM inference servers on Kubernetes.

Directories

• cpu-vllm/: vLLM serving google/gemma-2-2b-it on a local KinD cluster (CPU-only). Good for learning and quick iteration.
• gpu-vllm/: vLLM serving google/gemma-4-12b-it on an existing Kubernetes cluster with an NVIDIA GPU node.
• gpu-vllm-tp-single-node/: vLLM serving meta-llama/Llama-3.3-70B-Instruct sharded across 2 GPUs on a single node via tensor parallelism (TP=2).
• observability/: install kube-prometheus-stack and wire up the NVIDIA DCGM Grafana dashboard so gpu-vllm/ requests are visible in real time.

Each directory has its own README.md with prerequisites, deploy steps, and gotchas, and a run.sh that wires everything up.