investigate: test_txn_commit_wal_crash_recovery EAGAIN flake under parallel CI suite

[pilotspacex-byte]

Context

`test_txn_commit_wal_crash_recovery` (tests/txn_kv_wiring.rs:1025) intermittently fails on BOTH Linux and macOS CI runners with:

• Linux: `connect failed: Resource temporarily unavailable (os error 11)` (EAGAIN)
• macOS: `connect failed: Connection reset by peer (os error 54)`

Confirmed pre-existing (independent of PR #129):

• Run 26787786728 (dependabot bump tokio, June 1, off main) shows byte-identical EAGAIN failure
• Commit 98b578c (May 23) documents the test as a "long-standing flake"
• Commit c2a154a (May 25, PR chore(test): harden test_txn_commit_wal_crash_recovery flake diagnosis #118) calls it "the dominant CI flake on macOS + Linux"

Symptom

Phase-1 server logs clean startup ("AOF enabled", "Listening on 127.0.0.1:PORT", "WAL v3 writer initialized") but the test client's TCP connect to localhost returns EAGAIN/ECONNRESET for the entire 60s deadline.

Hypothesis

CI runner resource pressure after ~10 minutes of the 2814-test parallel suite — ephemeral port depletion, accept-queue saturation, or scheduler contention. The local test passes consistently (0.19s on unloaded Mac).

Investigation needed

1. Run with `--test-threads=1` to isolate from concurrent test resource contention.
2. Add `SO_REUSEADDR` to the test harness so port reuse isn't blocked by TIME_WAIT.
3. Strace the spawned moon process during a failing run to see if accept() is actually being called.
4. Consider gating the test behind `#[ignore]` and running it serially in a dedicated CI job (like crash_matrix_per_shard_aof was wired in FIX-W1-3).

References

• Triage workflow: PR fix(persistence): multi-shard AOF gate + per-shard AOF foundation (Option B step 1) #129 wtz0a5hme (diagnosis A)
• Prior commits: 98b578c, c2a154a, and others attempting to harden the test

author: Tin Dang

[pilotspacex-byte]

Closing — fixed by commit 3e3b55ab (PR #284, task #18, "TOCTOU-safe shared
port/spawn helpers across all 33 server-spawning suites"), which rewired
ChildGuard::spawn in tests/txn_kv_wiring.rs onto common::spawn_listening,
eliminating the bind-port TOCTOU race that caused the EAGAIN flake. The
commit body explicitly names this test's crash-recovery path.

Reopen if the flake recurs under the current harness.

Author: Tin Dang tindang.ht97@gmail.com