c10k hardening — remaining Medium/Low findings (D3, D4, E1, E3, E4, F1–F6)

[TinDang97]

Tracking issue for the unaddressed findings from the c10k hardening review
(tmp/C10K-HARDENING-REVIEW.md, 30 findings, 2026-07-30). Every 🔴 Critical and
🟠 High finding has now merged; what remains here is all 🟡 Medium / ⚪ Low.

Landed (for context)

• Cluster A (A1–A6): fix(blocking): stop cross-shard BLPOP from destroying the next push (c10k A2/A3/A4/A5/A6) #426, fix(blocking): reap clients that disconnect while blocked (c10k A1) #428
• Cluster B (B1–B5): fix(security): ACL gate before privileged intercepts, fail-closed unknown users, real CLIENT KILL USER (c10k cluster B) #430
• Cluster C (C1–C3): fix(conn): c10k cluster C — bound reply writes, query buffers, and maxclients rejections (+ allocator observability) #431
• D1: fix(conn): stop timeout N from silently disabling the c1M connection park (c10k D1) #427 · D2: fix(conn): one byte no longer hides a connection from the idle sweep (c10k D2) #434 · E2: fix(shard): FLUSHALL inside MULTI/EXEC clears every shard, not just one (c10k E2) #437
• E1 · E3 · E4 · D3-lazy: fix(shard): bounded cross-shard fan-out + reply awaits (c10k E1/E3/E4), lazy rehydrated buffers (D3) #441 (2026-08-07 — bounded fan-out pushes + bounded reply awaits + lazy rehydrated buffers; see the 2026-08-07 comment for D3's refined scoping and the deliberately-rejected resume-rate gate)

Remaining

Cluster D — park-design robustness

• 🟡 D4 (routing F7) — Migration races the pipelined batch tail. Gate checked mid-batch, migration executes at batch end with no re-check; MigratedConnectionState carries no command_queue/in_multi/subs/tracking. […GETs, MULTI, SET] → migrates with in_multi=true, queued txn discarded, tail SUBSCRIBE orphaned. Fix: re-evaluate the gate at the migration point.

Cluster F — lifecycle / shutdown / invariant hygiene

• 🟡 F1 (conn#8) — Graceful shutdown doesn't drain: shard returns from run, drops the runtime, every pending conn task dropped; clients see truncated replies, not clean FIN. Fix: bounded drain phase + shutdown arm on the stage-1/2 read parks.
• ⚪ F2 (conn#9 / sec L1) — kill_clients fd-liveness invariant no longer literally holds for parked conns (drop order = capture order). Not exploitable today; one refactor (F1's drain) from a shutdown(2)-on-reused-fd. Fix: wrap {guard,stream} with a hand-written guard-first Drop.
• ⚪ F3 (routing F10) — Central-accept loop head-of-line blocks on one wedged shard's full 4096-conn channel. Central-accept only (non-Linux/TLS/tests). Fix: try_send + round-robin next shard.
• ⚪ F4 (routing F12) — Migration fd lifecycle: raw i32 payload with no Drop leaks the fd on shutdown/undelivered; resumed-parked conns permanently can_migrate:false.
• ⚪ F5 (sec L3) — Migrated ConnectionContext carries requirepass:None (inert today, loaded gun for future auth re-derivation).
• ⚪ F6 (sec L2) — Unauthenticated conns are parkable with no auth deadline (park a maxclients-worth of silent conns at 3.3 KB each).
• ⚪ conn-secondary — kept_registration miss-arm counter-leak (fail-open); is_sweep_cancel matches bare errno 125 (non-sweep ECANCELED misread).

Test hygiene (found while validating #441)

• ⚪ pid-only --dir names race the moon.lock instance lock under parallel cargo-test. Fixed in pubsub_kv_ordering + pubsub_multi_channel_acl (fix(shard): bounded cross-shard fan-out + reply awaits (c10k E1/E3/E4), lazy rehydrated buffers (D3) #441); same pattern remains in info_memory_allocator_pagecache, vector_exact_rerank, vector_db_isolation, memory_doctor_response, memory_prometheus_kinds, tls_park_keyupdate.

Notes

• No Critical/High remains; these are defense-in-depth + correctness-under-saturation.
• Suggested next wave: D4 → F1/F2 (F1 and F2 interact — the drain refactor is what makes F2 exploitable, so land them together) → F3–F6 + conn-secondary.
• Full detail + line refs in tmp/C10K-HARDENING-REVIEW.md.

[TinDang97]

E-cluster + D3 update (2026-08-07)

PR #441 implements E1, E3, E4, and D3's lazy-buffer lever:

• E1 (cross-shard PUBLISH drop-on-full): all 3 sites → push_with_backpressure, loud give-up (moon_xshard_fanout_drop_total).
• E3 (SCRIPT LOAD fan-out drop): both sites → shared script_fanout_bounded.
• E4 (unbounded reply awaits): all 5 sites bounded at the coordinator's 30s XSHARD_REPLY_TIMEOUT. Slotted-dispatch expiry is fatal-per-connection by design — the per-connection ResponseSlot is reused across batches, so an abandoned await could hand a late fill to the next batch as its own reply.

D3 refined scoping (investigation on merged main): two of the three D3 sub-risks were already structurally mitigated by the merged campaign, narrower than this issue's original text:

1. Kill-burst herd: fleet-wide timeout expiry / CLIENT KILL wakes close in the park watcher itself (conn_accept.rs) — no handler rehydration at all.
2. Park-thrash: re-parking is sweep-driven under --conn-park-secs, so per-conn cycle rate is capped at 1/threshold; the review's 16k cycles/s figure was the 1M-conn aggregate (~µs of spawn work each).
3. Genuine-traffic wake burst: the residual gap. PR fix(shard): bounded cross-shard fan-out + reply awaits (c10k E1/E3/E4), lazy rehydrated buffers (D3) #441 takes the lazy-buffer lever (rehydrated handlers start at 512 B ×3 instead of 8 KiB ×3, grow on first real traffic). A resume-rate gate that queues real client requests to smooth RSS was evaluated and deliberately not taken — it trades request latency for memory with no demonstrating workload. Revisit only if a production trace shows a synchronized-wake RSS spike that the lazy buffers don't absorb.

Remaining open in this issue after #441: D4 (migration vs pipelined batch tail), E2 follow-ups if any, F1–F6, conn-secondary items.

[TinDang97]

D4 landed in #444 (merge 8e2fe8fc), which grew into a batch-tail hardening wave — writing the D4 tests surfaced a pre-existing Critical:

• Critical (new finding, fixed): any early-flush command (SUBSCRIBE/PSUBSCRIBE/blocking/PSYNC) mid-batch with pending remote-slotted commands flushed placeholder responses and cleared responses while remote_groups still held indices — phase-2 drain OOB → shard panic → whole-process abort, remotely triggerable on --shards ≥ 2. Fixed by deferring the early-flush command + unconsumed tail (lossless RESP re-encode) to the next batch iteration.
• D4: ConnectionState::migration_eligible() now gates the batch-end migration execute point as well as the latch (in_multi / cross-txn / subs / tracking / saw_replconf). Latch stays armed; migrates at first clean batch end.
• Migrated-connection carried remainder no longer stalls (resumed handler arms carried_input).
• Parsed batch tail after a SUBSCRIBE/blocking break is carried, not dropped.

Next up per the agreed order: F1+F2 together (graceful-shutdown drain + guard-first {guard,stream} drop). F1 red leg already proven on Linux io_uring: 19/50 BLPOP-blocked clients lose their shutdown reply pre-fix (macOS passes only by scheduling luck).

Also noted while landing: parked_idle_parity::parked_connection_visible_and_killable has a pre-existing assert-too-soon race (CLIENT LIST checked before the killed handler task gets scheduled to deregister) — fired 3/3 on a loaded 2-vCPU runner, passes solo. Will fix with a deadline-poll in the test-hygiene sweep.

[TinDang97]

F1 + F2 landed in #445 (merge 2bf488f6), plus a third bug the red test flushed out:

• F1 (conn#8): shards now run a bounded drain (5 s ceiling) before persistence teardown — parked stage-1/2 reads woken via cancellers, token arms cover tracking/subscriber/blocking parks, per-thread live-task counter as stop condition. Pre-fix Linux io_uring lost 19/50 blocked clients' shutdown replies; post-fix 50/50 on both runtimes.
• F2 (conn#9/sec L1): ParkedSession wraps the watcher's {guard, stream} — declaration-order drop (guard deregisters before fd close), contract documented + unit-pinned. kill_clients invariant restated with its owners.
• New find (tokio-only, same class): central-listener connections were io-bound to the MAIN runtime's driver; their writes died with main's runtime regardless of shard draining (~half of all tokio conns via SO_REUSEPORT split, "A Tokio 1.x context was found, but it is being shutdown"). Forwarded streams now re-register with the owning shard's driver at spawn.

Review round: drain rescan backoff (2 ms → 50 ms after 5 ticks); CodeRabbit's seal-intake point is half-tracked here — the queued-MigrateConnection fd half is F4, which is next. Proceeding with F3–F6 + conn-secondary.