feat(loadgen): zainod gRPC concurrency/load generator + multi-RPC driver

[nachog00]

What

Adds loadgen — a concurrency/load generator for a running zainod — and extends ztest's load-test driver to drive more than one RPC.

• crates/loadgen — a runnable front-end over ztest::loadtest. Attaches to a zainod gRPC endpoint (--target), fans out N per-task connections driving a chosen RPC (--rpc block-range|latest-block|block), and reports hdrhistogram percentiles, throughput, errors, and chain-link oracle violations. Runs locally or as an in-cluster k8s Job. Ships a Containerfile, k8s/job.yaml, scripts/sweep.sh, and a README with an architecture diagram.
• src/loadtest/{scenario,driver}.rs — LoadDriver now dispatches per-op from the scenario (Scenario::op() → Op) instead of hardcoding GetBlockRange. Adds LatestBlockPoll and BlockPoll scenarios; the oracle runs only for block-bearing ops (single block for GetBlock, skipped for GetLatestBlock). DiffLoadDriver is left block-range-only.

How to audit / run

See crates/loadgen/README.md (architecture diagram + flags + output format). In-cluster:

kubectl -n <zaino-ns> apply -f crates/loadgen/k8s/job.yaml
kubectl -n <zaino-ns> logs -f job/loadgen

Connection sweep: crates/loadgen/scripts/sweep.sh <zaino-ns> "1 4 8 16 32 64 128" 20

Findings so far

Against 0.7.0-rc.1 (fetch backend, mainnet, non-ephemeral — real finalised state), in-cluster:

• Per-RPC spread ≈ 4 orders of magnitude at 64 clients: GetLatestBlock ~265k ops/s (p99 0.6ms), GetBlock ~6.4k, GetBlockRange ~122 ops/s (p99 ~3s). The fetch backend's single upstream client to zebra is the entire GetBlockRange bottleneck.
• GetBlockRange saturates by ~64 clients and collapses at 128 (throughput drops below its 4-client level). 0 correctness violations throughout, including the collapse.
• ZAINO_EPHEMERAL_FINALISED_STATE inflates read numbers ~3×/~50× — not representative; the numbers above are from a non-ephemeral node.

Report (Markdown + Mermaid): https://gist.github.com/nachog00/fe8d4f9b0ad00716113976d2d17ffeef

Notes / follow-ups

• Measurement, not a calibrated SLO (the node isn't CPU-pinned) — trust the shapes and the backend/version differential, not raw absolutes.
• Follow-ups: DiffLoadDriver multi-op + parity; GetTreeState / GetMempoolStream (need the block-list Observed generalized); a loadgen sweep/report subcommand emitting a versioned JSON schema for GitHub reporting; a fetch vs state backend A/B.

[nachog00]

@elicbarbieri I bet this use mode doesn't converge with the ergonomic cli based interaction for ztest that you have been crafting, i just needed a one off bin to run on tekau that would let me drive zaino with these clients... how do you feel about it? did I miss a cleaner approach?