client-compat harness: warn on stale MOON_BIN — a days-old binary reports a confident, false result

[TinDang97]

Problem

scripts/test-client-compat.sh defaults MOON_BIN to "this checkout's release build" = target/release/moon. In an everyday working tree that binary is stale — normal development builds release-fast, and target/release/moon can be days old (it is explicitly quarantined in this repo). The harness reports the resulting divergences as ordinary FAILs with no indication that the binary under test is not the code under test.

Observed cost

2026-08-10, on a tree where the fix was already applied and green:

PASS=94  FAIL=32  WAIVED=26

The failure list looked like a broad regression across unrelated categories. The tell was subtle: one entry reported

FAIL [type] type_sismember_is_integer: integer != boolean
  REDIS: b':1\r\n'
  MOON:  b'#t\r\n'

…from int_to_bool, a function deleted hours earlier. Same tree, MOON_BIN pinned:

PASS=128 FAIL=0 WAIVED=24

A two-day-old binary produced a confident and entirely false compatibility picture. For a harness whose whole purpose is to be the trustworthy oracle-backed verifier, that is the one failure mode it must not have.

Proposed fix

1. Print provenance in the header, always — resolved MOON_BIN path, mtime, and --version. Cheap, and makes the mistake self-diagnosing.
2. Warn loudly (or refuse under --strict) when the binary is older than the newest file under src/. A stale binary under --strict should be ERR_STALE_BINARY, consistent with the existing ERR_NO_ORACLE philosophy: "a differential harness with no oracle proves nothing, and a green skip would be a lie." A differential harness testing the wrong binary is the same lie with extra steps.
3. Consider defaulting to release-fast when it exists and is newer than release.

Related trap

./scripts/test-client-compat.sh | tail reports tail's exit status, not the harness's. Any CI/scripted use must capture the real code. Worth a note in the script header.

[TinDang97]

Related: the self-test has the same problem, one level up

scripts/client-compat/test_e2e.py::test_a_diverging_entry_exits_one_and_names_the_divergence asserts that a diverging entry exits 1 and is classified as type. To do that it needs a divergence — and it borrows a real, open defect as its fixture.

That means the test fails as a reward for fixing something. It has now rotated three times:

1. GET inside MULTI — retired by the v0.8.6 inline-GET hotfix (fix(security): ACL bypass + silent CLIENT TRACKING failure on the inline GET path #457)
2. SISMEMBER RESP3 Integer-vs-Boolean — retired by fix(protocol): RESP3 reply types match Redis, and stop depending on calling context #463; the test started failing 0 != 1 the moment the fix landed
3. COMMAND COUNT — current, open, owned by client-identity-introspection

Each rotation costs a CI-red debugging cycle for whoever fixes the next reply-type defect.

Why there is no easy fix today

I looked for a divergence that is permanent by construction, and there isn't one:

candidate	Moon	Redis 8.6.1	usable?
TXN.ABORT	-ERR unknown command	-ERR unknown command	no — Error on both, no TYPE divergence
MQ PUSH	-ERR wrong number of arguments	-ERR unknown command	no — same
HOTKEYS	*0	-ERR wrong number of arguments for 'hotkeys'	no — redis 8.6.1 implements hotkeys, so this is arguably a Moon arity defect someone will fix

Moon's proprietary commands all surface as an Error on both sides, which diverges in text but not in type.

Proposal

A test-only injection hook so the self-test fabricates its divergence instead of borrowing one — e.g. an entry field the runner honours only under a test flag, that rewrites one side's reply before comparison. Then the assertion tests harness behaviour (what it does when replies differ), which is what it is actually for, and it stops being coupled to the state of the product.

Grouping here rather than filing separately: this and the stale-MOON_BIN problem are the same theme — the harness cannot currently tell the truth about itself.