Skip to content

test-commands.sh aborts before its summary, and several rows have never passed #679

Description

@TinDang97

scripts/test-commands.sh cannot complete a run, and some rows have never passed

CLAUDE.md describes this script as "190 tests across 13 categories" and requires
new commands to add entries to it. In practice a full run aborts partway and
never prints its result summary, so nobody sees the totals — and several rows
have been failing since the day they were written.

Found while adding the DEBUG DIGEST rows for #636. Two of the aborts are
fixed in that PR because without them the script cannot report at all; the rest
are filed here rather than folded into an unrelated change.

Fixed in #636

  1. Four redis-cli -p "$PORT" sites (vector block, ~1480–1510) named a
    variable the script never defines — PORT_RUST is the real one; PORT only
    exists later as a loop variable in NUMERIC-07. Under set -euo pipefail the
    first one aborted the run, so MQ, txn_kv, eviction, benchmark and the summary
    never executed. Same class as fix(keyspace): RANDOMKEY samples the whole keyspace, not the serving shard (#629) #634.
  2. Two unguarded pkill in NUMERIC-07. pkill exits 1 when nothing
    matched, which under set -e ends the script — on a clean machine the first
    one fires before anything has been spawned.
  3. The two ports were hard-coded at 6399/6400; they now honour
    PORT_REDIS/PORT_RUST like the sibling test-consistency.sh. A squatter
    on 6399 silently makes the "expected" side of every row come from a second
    moon.

Still open

a. grep -P (8 sites, ~1937–2104) is not portable. On this macOS host
grep is ugrep 7.8.4 and rejects it:

$ printf 'open\n5\n' | grep -Pzo "(?s)open.*5"
grep: invalid option -- P          (exit 2)

Every AGG-/TAG-/NUMERIC-04 row built on it fails with got: 2 — that "2" is
grep's exit code leaking into the comparison, not a moon answer. These rows pass
only on a host with GNU grep.

b. N1=$(... | grep '^n:' | sort) aborts the script under pipefail.
NUMERIC-07 is where the run currently stops: when the search returns nothing,
grep exits 1, pipefail propagates it, and set -e kills the run before the
summary. A test that finds no results should FAIL its row, not end the script.

c. Rows that have never passed. FT.CREATE ... SCHEMA embedding VECTOR FLAT
expects OK, but moon accepts only HNSW:

CMD:      FT.CREATE myidx ON HASH PREFIX 1 doc: SCHEMA embedding VECTOR FLAT 6 DIM 4 ...
EXPECTED: OK
GOT:      ERR expected HNSW algorithm

ERR expected HNSW algorithm has been in ft_create.rs since #27 and the row
since #35/#144, so this has failed from the start — the five FT rows that depend
on myidx existing (FT.INFO, FT.SEARCH, FT.DROPINDEX, …) fall over behind
it. Either moon should accept FLAT or the row should use HNSW; right now it
is asserting a behaviour that was never implemented.

d. NUMERIC-07 spawns servers into the repo root.

./target/release/moon --port 6411 --shards 1 --protected-mode no &

No --dir, so moon uses the CWD — it writes persistence into the checkout and
reloads whatever a previous run left there (the main server logs
--dir not set and existing moon data found in the current directory for the
same reason). It also hard-codes ./target/release/moon instead of honouring
MOON_BIN, so it can silently run a binary from another branch.

e. line 330: echo: write error: Broken pipe intermittently turns the
COMMAND row into a failure — a head closing the pipe early. Passes when the
connection category is run alone, fails in a full run.

Why this matters

A harness that stops early and prints no totals reads exactly like a harness
that passed. Suggested minimum: make the script assert its own completion (a
summary line is mandatory), and treat a missing summary as failure in any
wrapper that calls it.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions