Replicas cannot apply streamed SWAPDB — records silently no-op (master/replica divergence)

[pilotspacex-byte]

Found by the adversarial review of PR #385 (#133). Pre-existing, NOT introduced there — but #385's scope was cut to exclude it, so tracking here.

Defect

replication/apply.rs::apply_local has no special case for SWAPDB: it falls through to generic cmd_dispatch, and command/mod.rs hard-errors SWAPDB with "ERR SWAPDB must be issued at the connection handler level". warn_on_error only logs; the outcome still counts as applied — no resync is triggered. Every SWAPDB record a replica receives silently no-ops.

Today the remote legs of multi-shard SWAPDB already fan the record out via wal_append_and_fanout (backlog + offset + live replica send) — so a multi-shard master with a replica attached diverges on every SWAPDB: master swaps, replica keeps the old databases, offsets advance as if applied. Recovery replay is NOT affected (persistence/replay.rs::replay_command special-cases SWAPDB correctly — different code path).

Scope of fix

1. Replica-side: special-case SWAPDB in apply_local (swap the applying shard's local databases — per-shard streams mean each replica shard applies its own record exactly once; verify parity semantics for the R2 multi-shard PSYNC merge).
2. Master-side local leg: PR fix(shard): multi-shard SWAPDB — durable + replica-visible before +OK (#133) #385 deliberately does NOT record the coordinator shard's SWAPDB on the replication plane (the reverted record_local_swapdb_repl did backlog/offset bookkeeping BEFORE the durability gate, which diverges master/replica when the local AOF/fsync aborts — see PR fix(shard): multi-shard SWAPDB — durable + replica-visible before +OK (#133) #385 review, Vector 2). When re-adding it: record AFTER local durability is confirmed and the local swap is applied, or make the abort path retract/never-send.
3. Tests: end-to-end master→replica SWAPDB (single- and multi-shard), plus an injected local-fsync-failure case asserting no replica-side swap.

References

• PR fix(shard): multi-shard SWAPDB — durable + replica-visible before +OK (#133) #385 adversarial review, Vectors 2 and 5 (full analysis in the PR comments)
• replication/apply.rs (apply_local fallthrough), command/mod.rs SWAPDB early-return
• persistence/replay.rs::replay_command (the correct recovery-side handling to mirror)

author: Tin Dang