test flake class: 23 test files sleep a fixed 250ms instead of probing readiness, causing 'Connection refused' failures

[TinDang97]

Summary

23 integration-test files spawn the server on a background thread and then wait for it with
a fixed sleep(250ms) instead of probing readiness. When 250 ms is not enough — a loaded
machine, a cold page cache, high suite parallelism — every test in the file fails with:

called `Result::unwrap()` on an `Err` value: Connection refused (os error 61)

This is the dominant source of ci-local macOS-host-leg failures I have seen: the leg failed
4 out of 4 runs, each time on a different test, and at least two of those
(ft_search_temporal_parity, txn_ft_search_snapshot) are files in this set.

The pattern

tests/ft_search_temporal_parity.rs:288:

std::thread::spawn(move || {
    let rt = tokio::runtime::Builder::new_current_thread().enable_all().build()...;
    rt.block_on(async {
        if let Err(e) = listener::run_with_shutdown(config, listener_cancel).await { ... }
    });
});

tokio::time::sleep(std::time::Duration::from_millis(250)).await;   // <-- the bug
(port, token)

async fn connect(port: u16) -> redis::aio::MultiplexedConnection {
    let client = redis::Client::open(format!("redis://127.0.0.1:{port}")).unwrap();
    client.get_multiplexed_async_connection().await.unwrap()   // line 313 — panics here
}

Nothing observes that the listener is bound. 250 ms is a guess, and the whole file's tests
share the helper, so a slow start takes all of them down together (observed: 3 of 4 failed at
the identical line, which is what distinguishes this from a race).

Evidence

run	branch	macOS-host failure
1	docs-only	manifest::tests::test_overflow_compaction_bounds_growth (→ #750)
2	docs-only	sigterm_clean_exit_shards_4_held_conn (→ #751)
3	skill-only	oom_bypass_closure, txn_ft_search_snapshot ← this class
4	docs-only	ft_search_temporal_parity ×3 ← this class

All pass in isolation (ft_search_temporal_parity 0/3 failed standalone). Run 4 was on an
otherwise-quiet machine (load 3.9), so this is not purely a load artifact — the margin is just
too thin.

Both branches involved have an empty git diff main -- src/ tests/, so none of this is
change-induced.

Why these files are affected and others are not

The repo already solved this. tests/common/mod.rs provides reserve_port() (dedupes ports
intra-process) and spawn_listening_guarded() (binds, verifies, respawns on a dead child).
Files that use them are not in this failure set. The 23 affected files do not import
tests/common at all — they spawn listener::run_with_shutdown in-process, which
spawn_listening_guarded does not cover because it wraps a subprocess Child.

Suggested fix

Add an in-process equivalent to tests/common — poll-connect until the port accepts, with a
generous deadline and a fail-fast error, mirroring wait_for_ready's shape:

pub async fn await_listening(port: u16, deadline: Duration) -> Result<(), String> {
    let start = Instant::now();
    loop {
        if TcpStream::connect(("127.0.0.1", port)).await.is_ok() { return Ok(()); }
        if start.elapsed() > deadline {
            return Err(format!("port {port} never accepted within {deadline:?}"));
        }
        tokio::time::sleep(Duration::from_millis(10)).await;
    }
}

then replace the 23 sleep(250ms) sites with it. Mechanical, and it removes a whole failure
class rather than widening one timeout.

Bumping 250 ms to a larger constant would paper over it and make the suite slower for
everyone — the deadline should be an upper bound that is rarely reached, not the wait itself.

Related

• crash_matrix_per_shard_aof: unique_port()+N uses unverified ports, causing 'Connection refused' flakes #489 — same Connection refused symptom, subprocess/port-allocation side
• flake: manifest commit tests can observe another test's TEST_INJECT_PERSIST_ERROR (process-global knob, guard not taken by readers) #750, flake: sigterm_shutdown server shuts down gracefully during startup, before readiness (exit 0, not a port conflict) #751 — the other two macOS-host failures from the same sessions (different causes)