fix(vector): a truncated FT.CREATE no longer aborts the whole process (#681) - #682
Conversation
…#681) `FT.CREATE idx ON HASH PREFIX 1 d: SCHEMA v VECTOR HNSW` -- argv cut off right after the algorithm keyword -- read one past the end of `args` and panicked. The panic ran on a shard thread, and moon deliberately escalates a shard panic to a whole-process abort rather than serve on with a dead shard, so one short line from any client took the server down: every database, every other connection. No auth, no large payload. The fix is one bounds check returning `ERR invalid param count`, the error an unparseable count already produced, so the two ways of failing to supply a count are indistinguishable to a client. There is no redis oracle for the string -- the redis-server checked against has no query engine, so FT.CREATE is `unknown command` there. Scope was measured, not assumed. The first guess was that every `*pos += 1; args[*pos]` pair in the function was a separate crash; six truncation shapes probed against freshly spawned, listener-PID-checked servers said otherwise. The parameter loop guards both ends (`*pos + 1 < param_end && *pos + 1 < args.len()`), so every value read was already safe -- the count read was the one unguarded site. A second test pins those seven neighbours so a future edit to the loop condition cannot quietly reopen the hole one keyword further in. FT.CREATE argument parsing had no fuzz target for its whole life, which is how a one-line remote crash survived in it. The new target took three designs, each rejected by measurement against a deliberately un-fixed parser rather than by inspection: 1. Fully-arbitrary argv -- 1.2M execs, found nothing. `ft_create` demands the preamble `idx ON HASH PREFIX 1 d: SCHEMA v VECTOR` before the vector parser is reached; random mutation does not synthesize a nine-keyword sequence, so it fuzzed the preamble and never got past it. 2. Valid skeleton + byte-level tail -- 871K execs, still nothing. Reaching the parser is not enough: the crash needs the literal ASCII `HNSW` in argv, and inventing a specific four-byte string by mutation is a 2^32 search. 3. Skeleton + one input byte selecting one argv element from the vocabulary the parser branches on -- finds the panic, writes a reproducer artifact. Verified in both directions: the target crashes the un-fixed parser at ft_create.rs:561 with `index out of bounds: the len is 10 but the index is 10`, and that same saved artifact executes clean against the fix. Listed in BOTH matrices in fuzz.yml -- a target that exists but is not listed never runs (#576). Also verified end-to-end on a live server with the listener PID checked against the spawned process: the crashing command answers `ERR invalid param count`, the server stays up, zero panics in the log, and a well-formed FT.CREATE still returns OK with a queryable index. Gates: cargo fmt --check, clippy --all-targets -D warnings on default AND runtime-tokio feature sets, 4/4 new unit tests. author: Tin Dang
|
ⓘ Qodo reviews are paused because the subscription is no longer active. Ask your workspace admin to reactivate the subscription to resume reviews. Manage billing |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (5)
Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review. 📝 WalkthroughWalkthroughThe FT.CREATE vector-field parser now rejects a missing HNSW parameter count instead of panicking. Unit tests cover truncated inputs, and a new ChangesFT.CREATE parser safety
Estimated code review effort: 3 (Moderate) | ~20 minutes Merge Risk: ⚪ Minimal · up to The PR prevents truncated FT.CREATE commands from terminating the server while preserving valid command behavior; no actionable merge-blocking risk remains after normal checks and review. Sequence Diagram(s)sequenceDiagram
participant libFuzzer
participant ft_create
participant vector_store
participant text_store
libFuzzer->>ft_create: generated FT.CREATE argument frames
ft_create->>vector_store: initialize vector store
ft_create->>text_store: initialize text store
ft_create-->>libFuzzer: process survival or failure
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…tals (#679) (#684) The suite CLAUDE.md points at for every new command had never printed a result. It died partway through, silently, and the operator saw a truncated log instead of a summary. It now finishes: 504 rows, 478 passing. The 26 remaining failures are pre-existing and are filed as #683 -- they became visible for the first time because the run now reaches them. Nothing here weakens an assertion to get a green. Every abort was the same bash shape: a command whose non-zero status `set -euo pipefail` converts into a silent exit. grep exits 1 when it matches nothing, lsof exits 1 when a port is free, pkill exits 1 when nothing matched, and a shell function whose last statement is a false `if` returns 1 as well. Because that failure prints nothing at all, each instance hid the next -- which is why this took several rounds, and why two of the guards below are for aborts introduced by earlier ones in this same commit. Guarded: * 26 command substitutions ending in grep. The reported NUMERIC-07 site was one of a class, so the class is fixed rather than the instance. * 6 raw redis-cli pipelines and all four client wrappers, so a dead server yields failing rows plus a summary instead of a truncated log. Nothing in the file branches on those wrappers' exit status (checked: no `if mcli` and no `mcli ... &&` anywhere), so `|| true` costs no signal. * `cargo build ... 2>/dev/null`, which threw away the reason a build failed and left the log reading "Building moon..." and nothing else. * the cleanup trap, which returned its last kill's status rather than the script's -- reporting a clean run as a failure. Three defects that produced wrong results rather than aborts: * `grep -Pzo "(?s)A.*B"` at 13 call sites is GNU-only. On a macOS host grep is ugrep, which rejects -P and exits 2; since those rows compare output rather than status, that 2 was being reported as moon's answer. Replaced with a portable spans() helper. * No --dir, so moon treated the CWD as its data dir: the suite wrote appendonlydir/ and moon.lock into the repo root and reloaded the previous run's FT index definitions, so a second run failed with "Index already exists". Each run now gets a fresh mktemp dir, removed on exit. * No port pre-flight. A leftover server from an unrelated run answers and every row silently compares against it. Not hypothetical: it produced a full run of MOONERR diskfull failures traced to another session's moon on the port. The suite now refuses to start on an occupied port and names the holder. The `FT.CREATE ... VECTOR FLAT` row expected OK, but moon has only ever implemented HNSW (ERR expected HNSW algorithm, in ft_create.rs since #27), so it had failed from the day it was written and took four dependent rows with it. It now builds an HNSW index, and the FLAT gap is asserted explicitly instead of hiding inside a row that expected success. Added a regression row for #681 asserting the server is still alive after a truncated FT.CREATE. Proven in both directions: it fails against a pre-#682 binary and passes after. CLAUDE.md's "190 tests" was stale by more than half. Fixes #679 Refs #681, #682, #683 author: Tin Dang
Closes #681.
The bug
parse_vector_field_paramsbounds-checks the algorithm keyword, advances pastit, then reads the parameter count with no second check. When
HNSWis thelast argument,
*posbecomesargs.len(). The panic is on a shard thread andmoon escalates that to a process abort by design, so the blast radius is the
whole server — every database, every other connection. No auth, no large
payload, one short line.
Present on
mainsince the file was written (#27).Scope was measured, not assumed
My first read was that every
*pos += 1; args[*pos]pair in the function wasits own crash — about a dozen. That was wrong. The parameter loop guards both
ends (
*pos + 1 < param_end && *pos + 1 < args.len()), so every value readis safe; only the count read was unguarded.
Six truncation shapes, each against a freshly spawned server whose listener PID
was checked against the process I started:
... VECTOR HNSWft_create.rs:550:39... HNSW 6 TYPE... HNSW 6 TYPE FLOAT32 DIM... HNSW 6 TYPE FLOAT32 DIM 4 DISTANCE_METRIC... HNSW 6 TYPE FLOAT32 DIM 4 MA second test pins those seven neighbours as already-bounded, so a future edit
to the loop condition cannot quietly reopen the hole one keyword further in.
The fix
One bounds check returning
ERR invalid param count— the error anunparseable count already produced, so the two ways of failing to supply a
count are indistinguishable to a client. There is no redis oracle for the
string: the
redis-serverchecked against has no query engine, soFT.CREATEis
unknown commandthere.The fuzz target took three designs
FT.CREATEargument parsing had no fuzz target for its whole life, whichis how a one-line remote crash survived in it. Each design was rejected by
measurement against a deliberately un-fixed parser, not by inspection:
ft_createdemandsthe preamble
idx ON HASH PREFIX 1 d: SCHEMA v VECTORbefore the vectorparser is reached, and random mutation does not synthesize a nine-keyword
sequence. It was fuzzing the preamble and never getting past it: builds,
runs, reports clean, cannot see the bug it exists for.
parser is not enough; the crash needs the literal ASCII
HNSWin argv, andinventing a specific four-byte string by mutation is a 2^32 search.
the parser branches on — finds the panic and writes a reproducer.
Verified in both directions:
ft_create.rs:561withindex out of bounds: the len is 10 but the index is 10and saves anartifact;
Executed ... in 46 ms).Listed in both matrices in
fuzz.yml— a target that exists but is notlisted never runs (#576) — and confirmed not caught by the bare-
fuzzgitignore trap.
Verification
failed with the panic itself, not an assertion mismatch.
answers
ERR invalid param count, the server stays up, zero panics in thelog,
FLATkeeps its own distinct message, and a well-formedFT.CREATEstill returns
OKwith an index queryable viaFT.INFO.cargo fmt --check,clippy --all-targets -D warningson both thedefault and
runtime-tokiofeature sets.Summary by CodeRabbit
Bug Fixes
FT.CREATEcommands contain truncated or incomplete vector-search arguments.ERR invalid param counterror.Tests
FT.CREATEarguments.