client_tracking_invalidation: multikey DEL/MSET second-key invalidation push intermittently not delivered (~25-40%/run)

[TinDang97]

Split out of the #446 gate runs (2026-08-07) so the flake has an owner instead of living in test-rerun folklore.

Symptom. On the Linux VM (monoio release-fast), tests/client_tracking_invalidation.rs fails intermittently — observed across 7 runs at roughly 25–40%:

• multikey_del_invalidates_every_key: BOTH deleted keys must be invalidated, not just the first (a=true, b=false, got: ">2\r\n$10\r\ninvalidate\r\n*1\r\n$4\r\nct:a\r\n") — the push for the SECOND deleted key never arrives.
• mset_invalidates_every_second_arg_key: tracking client of ct:m2 must be invalidated by MSET (got: "") — no push at all for the second MSET key.
• nontracking_writer_invalidates_cross_connection has failed with the same shape.

Not a #446 regression. Merge-base A/B on the same VM: unmodified main (2bf488f) failed 1/3 runs, the F3–F6 branch 2/3, same signature — pre-existing. Related history: the 4-shard variant of this class is documented as "push lands ~75%/attempt" (PR #247 era), where the then-fix was to assert a majority rather than all-N. These single-run asserts are about per-KEY completeness on one connection, which SHOULD be deliverable — a real RESP3 tracking client would equally miss the second key's invalidation, so this smells like a delivery race in the multikey write→tracking-table fan-out rather than a test problem.

Where to look. Multikey write paths' tracked-key fan-out (TrackedWriteKeys through the dispatch legs), and the tracking push channel's delivery when two invalidation pushes race one park/read cycle on the subscriber conn.

Repro. orb run -m moon-dev → CARGO_TARGET_DIR=target-linux MOON_BIN=$PWD/target-linux/release-fast/moon cargo test --profile release-fast --test client_tracking_invalidation — fails within ~3 runs under sibling-suite load; less often solo.

[TinDang97]

Sharper characterization: at --shards 4 this is deterministic, not a 25–40% flake

Found while adding blocking-pop tracking coverage (#644 / PR #646). The consistency suite's existing cross-shard invalidation controls fail every run at --shards 4, on two different binaries, across four full suite executions:

  FAIL: tracking: ZINTERSTORE DEST invalidated [control]
  FAIL: tracking: SORT..STORE DEST invalidated [control]
  FAIL: tracking: SINTERSTORE DEST invalidated [control]
  FAIL: tracking: ZRANGESTORE DEST invalidated [control]
  FAIL: tracking: RENAME invalidates its source [control]

Same five, same order, on:

• moon @ 2b1f115d (pre-CLIENT TRACKING: every blocking pop (BLPOP/BRPOP/BLMPOP/BZPOPMIN/BZMPOP/BLMOVE/BRPOPLPUSH) invalidates nothing — client-side caches go permanently stale #644-fix), --shards 4
• moon @ PR fix(tracking): blocking pops must invalidate the key they served (#644) #646 head (post-fix), --shards 4, twice

and zero of them fail at --shards 1 (396/397, the only failure being ROLE on a master, which is #536).

So for the *STORE DEST and RENAME source cases the cross-shard invalidation push does not arrive at all at --shards 4 — it is not intermittent. That is a different defect from the one this issue describes (a multikey DEL/MSET second-key push arriving ~60–75% of the time), even if they share a root.

Two consequences worth acting on:

1. The 25–40% framing may be hiding a deterministic failure behind a probabilistic one. A test that retries, or an assertion relaxed to "majority delivers", will mask a push that never arrives — which is what happened here: these five rows have presumably been failing on every multi-shard run and reading as the known flake.
2. The remaining flaky case should be re-measured separately now that a deterministic one is known to be mixed in with it. routed_multi_exec_invalidates_tracking_client still needed 3 attempts to pass in a local ci-local run on an unrelated branch today, so the intermittent half is real too — but it should be characterized on its own.

PR #646 does not touch any of this; it is recorded here so the next person does not re-derive it.

[TinDang97]

Another instance, this time on the monoio runtime (VM, io_uring live) rather than tokio — routed_multi_exec_invalidates_tracking_client failed all 3 nextest retries in a ci-local run:

TRY 1 FAIL [24.987s] panicked at tests/client_tracking_invalidation.rs:446:9
TRY 2 FAIL [10.613s] panicked at tests/client_tracking_invalidation.rs:450:9
TRY 3 FAIL [ 2.364s] panicked at tests/client_tracking_invalidation.rs:208:5

Suite total: 6115 run, 6114 passed (3 flaky), 1 failed.

Two details that may help narrow it:

1. Each retry panicked at a different line (446 / 450 / 208), so it is not one deterministic assertion — it looks like which invalidation push goes missing varies per run, consistent with the ~25-40% per-key delivery loss this issue describes rather than a single broken path.
2. It passes in isolation: re-running just this test on the same VM immediately afterwards passed (46.7s). So the loss needs the full concurrent suite, not just multi-shard routing.

Environment: moon 0.8.7, moon-dev VM (Ubuntu 24.04, aarch64, io_uring), --profile release-fast, host under heavy external CPU load at the time. Seen on a branch with an empty git diff main -- src/ tests/, so it is not change-induced.