Support Opt-in FP8 KV Cache quantization with FA2-4 backend.

docs/runtime.md

@@ -304,6 +304,31 @@ zero watermark in non-incremental mode, see [scheduler admission](scheduler.md#a
 Pool sizing establishes physical capacity; scheduler admission decides how much
 of that capacity can safely be promised to new work.
 
+## FP8 KV cache
+
+`CacheConfig(kv_cache_dtype="fp8_e4m3")` (CLI `--kv-cache-dtype fp8_e4m3`) stores K and V as
+float8 e4m3 codes with one float32 scale per (block, layer, token, KV head) vector
+(`scale = amax / 448`). Values are quantized when written and dequantized by the attention
+backend when read, so a block costs `head_dim + 4` bytes per vector instead of
+`head_dim * element_size`: 51.6% of BF16 at `head_dim=128`, and automatic sizing or
+`--kv-cache-memory-bytes` admits about 1.94x as many tokens. The default `"auto"` stores the
+model dtype and leaves every existing path unchanged.
+
+- Supported: Torch (reference) and Triton attention; synchronous and asynchronous scheduling;
+  CUDA graphs (quantized writes are captured); LAST_EXITED finalize copies, including the
+  asynchronous finalize kernel, copy codes and scales; preemption snapshots carry the scales;
+  `KVCacheManager.read` returns dequantized values.
+- Triton's split-KV decode and packed paged-prefill kernels both apply the scales on load
+  (prefill dequantizes to the query dtype before `tl.dot`). Device-masked writes (speculative
+  verification, bucketed CUDA graphs) use the same fused writer with the row mask applied.
+- Triton writes with one fused quantize-and-store kernel per layer. The Torch backend quantizes
+  with Torch ops and is the reference; both writers produce the same codes and scales.
+- Rejected: FlashAttention backends (the `fp8_kv` backend capability; `auto` selects Triton)
+  and PD transfer, which does not yet move the scales.
+- Each layer writes the current token's K/V before attending, so attention reads the current
+  token in FP8 as well. Accuracy should be checked per model and workload; adaptive exits can
+  change where a token exits because the gate sees slightly different attention.
+
 ## Replay and validation commands
 
 Trace mode expects `depths_by_request`, with one depth per output: full prefill depth first, then each decode input's exit depth. Admission validates coverage and loop bounds. Request IDs must match the trace. The public engine replay fixes depth decisions; token generation still follows the configured sampler.

tests/test_cuda_graph.py

@@ -455,7 +455,8 @@ def test_bucketed_kv_graph_combines_active_rows_and_pipeline_write_mask():
 
     keys = torch.full((4, 1, 2, 1, 4), -1.0, device="cuda")
     values = torch.full_like(keys, -2.0)
-    cache = SimpleNamespace(key_cache=keys, value_cache=values)
+    # A BF16 pool has no FP8 scales, as KVCacheManager reports them.
+    cache = SimpleNamespace(key_cache=keys, value_cache=values, key_scale=None, value_scale=None)
     batch = SimpleNamespace(
         active_rows=torch.zeros((), dtype=torch.int32, device="cuda"),
         write_mask=torch.ones(5, dtype=torch.bool, device="cuda"),