You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
scripts/test-commands.sh cannot complete a run, and some rows have never passed
CLAUDE.md describes this script as "190 tests across 13 categories" and requires
new commands to add entries to it. In practice a full run aborts partway and
never prints its result summary, so nobody sees the totals — and several rows
have been failing since the day they were written.
Found while adding the DEBUG DIGEST rows for #636. Two of the aborts are
fixed in that PR because without them the script cannot report at all; the rest
are filed here rather than folded into an unrelated change.
Four redis-cli -p "$PORT" sites (vector block, ~1480–1510) named a
variable the script never defines — PORT_RUST is the real one; PORT only
exists later as a loop variable in NUMERIC-07. Under set -euo pipefail the
first one aborted the run, so MQ, txn_kv, eviction, benchmark and the summary
never executed. Same class as fix(keyspace): RANDOMKEY samples the whole keyspace, not the serving shard (#629) #634.
Two unguarded pkill in NUMERIC-07. pkill exits 1 when nothing
matched, which under set -e ends the script — on a clean machine the first
one fires before anything has been spawned.
The two ports were hard-coded at 6399/6400; they now honour PORT_REDIS/PORT_RUST like the sibling test-consistency.sh. A squatter
on 6399 silently makes the "expected" side of every row come from a second
moon.
Still open
a. grep -P (8 sites, ~1937–2104) is not portable. On this macOS host grep is ugrep 7.8.4 and rejects it:
Every AGG-/TAG-/NUMERIC-04 row built on it fails with got: 2 — that "2" is
grep's exit code leaking into the comparison, not a moon answer. These rows pass
only on a host with GNU grep.
b. N1=$(... | grep '^n:' | sort) aborts the script under pipefail.
NUMERIC-07 is where the run currently stops: when the search returns nothing, grep exits 1, pipefail propagates it, and set -e kills the run before the
summary. A test that finds no results should FAIL its row, not end the script.
c. Rows that have never passed.FT.CREATE ... SCHEMA embedding VECTOR FLAT
expects OK, but moon accepts only HNSW:
CMD: FT.CREATE myidx ON HASH PREFIX 1 doc: SCHEMA embedding VECTOR FLAT 6 DIM 4 ...
EXPECTED: OK
GOT: ERR expected HNSW algorithm
ERR expected HNSW algorithm has been in ft_create.rs since #27 and the row
since #35/#144, so this has failed from the start — the five FT rows that depend
on myidx existing (FT.INFO, FT.SEARCH, FT.DROPINDEX, …) fall over behind
it. Either moon should accept FLAT or the row should use HNSW; right now it
is asserting a behaviour that was never implemented.
d. NUMERIC-07 spawns servers into the repo root.
./target/release/moon --port 6411 --shards 1 --protected-mode no &
No --dir, so moon uses the CWD — it writes persistence into the checkout and
reloads whatever a previous run left there (the main server logs --dir not set and existing moon data found in the current directory for the
same reason). It also hard-codes ./target/release/moon instead of honouring MOON_BIN, so it can silently run a binary from another branch.
e. line 330: echo: write error: Broken pipe intermittently turns the COMMAND row into a failure — a head closing the pipe early. Passes when the connection category is run alone, fails in a full run.
Why this matters
A harness that stops early and prints no totals reads exactly like a harness
that passed. Suggested minimum: make the script assert its own completion (a
summary line is mandatory), and treat a missing summary as failure in any
wrapper that calls it.
scripts/test-commands.shcannot complete a run, and some rows have never passedCLAUDE.md describes this script as "190 tests across 13 categories" and requires
new commands to add entries to it. In practice a full run aborts partway and
never prints its result summary, so nobody sees the totals — and several rows
have been failing since the day they were written.
Found while adding the
DEBUG DIGESTrows for #636. Two of the aborts arefixed in that PR because without them the script cannot report at all; the rest
are filed here rather than folded into an unrelated change.
Fixed in #636
redis-cli -p "$PORT"sites (vector block, ~1480–1510) named avariable the script never defines —
PORT_RUSTis the real one;PORTonlyexists later as a loop variable in NUMERIC-07. Under
set -euo pipefailthefirst one aborted the run, so MQ, txn_kv, eviction, benchmark and the summary
never executed. Same class as fix(keyspace): RANDOMKEY samples the whole keyspace, not the serving shard (#629) #634.
pkillin NUMERIC-07.pkillexits 1 when nothingmatched, which under
set -eends the script — on a clean machine the firstone fires before anything has been spawned.
PORT_REDIS/PORT_RUSTlike the siblingtest-consistency.sh. A squatteron 6399 silently makes the "expected" side of every row come from a second
moon.
Still open
a.
grep -P(8 sites, ~1937–2104) is not portable. On this macOS hostgrepis ugrep 7.8.4 and rejects it:Every AGG-/TAG-/NUMERIC-04 row built on it fails with
got: 2— that "2" isgrep's exit code leaking into the comparison, not a moon answer. These rows pass
only on a host with GNU grep.
b.
N1=$(... | grep '^n:' | sort)aborts the script underpipefail.NUMERIC-07 is where the run currently stops: when the search returns nothing,
grepexits 1,pipefailpropagates it, andset -ekills the run before thesummary. A test that finds no results should FAIL its row, not end the script.
c. Rows that have never passed.
FT.CREATE ... SCHEMA embedding VECTOR FLATexpects
OK, but moon accepts only HNSW:ERR expected HNSW algorithmhas been inft_create.rssince #27 and the rowsince #35/#144, so this has failed from the start — the five FT rows that depend
on
myidxexisting (FT.INFO,FT.SEARCH,FT.DROPINDEX, …) fall over behindit. Either moon should accept
FLATor the row should useHNSW; right now itis asserting a behaviour that was never implemented.
d. NUMERIC-07 spawns servers into the repo root.
./target/release/moon --port 6411 --shards 1 --protected-mode no &No
--dir, so moon uses the CWD — it writes persistence into the checkout andreloads whatever a previous run left there (the main server logs
--dir not set and existing moon data found in the current directoryfor thesame reason). It also hard-codes
./target/release/mooninstead of honouringMOON_BIN, so it can silently run a binary from another branch.e.
line 330: echo: write error: Broken pipeintermittently turns theCOMMANDrow into a failure — aheadclosing the pipe early. Passes when theconnectioncategory is run alone, fails in a full run.Why this matters
A harness that stops early and prints no totals reads exactly like a harness
that passed. Suggested minimum: make the script assert its own completion (a
summary line is mandatory), and treat a missing summary as failure in any
wrapper that calls it.