Shard spins at ~266% CPU and stops answering commands during sustained ingest (HSET + FT.* writes)

[TinDang97]

Summary

During a sustained Lunaris ingest, a single Moon shard entered a CPU spin
(~266%) and stopped answering all commands — redis-cli PING blocks
indefinitely even though the TCP listener is still up — and made no further
persistence_tick checkpoint progress for 40+ minutes. The only recovery was
SIGKILL. One pathological ingest payload wedged the entire shard, not just the
offending request.

Environment

• moon release build, single shard (--shards 1)
• flags: --appendonly no --save "" --disk-free-min-pct 1, --protected-mode no, port 6381
• macOS (Darwin 24.6, arm64), host under heavy memory pressure at the time (swap ~87% full, ~2.5 GB free) — noting this as a possible contributing factor, not a confirmed cause.

Trigger

Ingesting the haystack for LongMemEval-S (longmemeval_s) question qid
1faac195 (dataset offset 57) — ~46 sessions / ~494 turns, written as ~46
documents through Lunaris ingest_episode. Each document = one atomic_write
(MULTI) carrying HSET hydration docs plus FT vector + keyword index upserts
(FT.*). Offsets 0–56 ingested cleanly; the wedge reproduced on this specific
question.

Observed

• Moon process pinned at ~266% CPU, sustained, for 40+ min (compute spin, not an idle wait).
• redis-cli -p 6381 PING blocks — no reply — while the listener still accepts connections.
• Last moon::shard::persistence_tick: Checkpoint complete log line ~40 min before kill, then total silence.
• The Lunaris client was blocked on the in-flight command. Lunaris bounds every Moon command with a 10 s per-command response timeout (LUNARIS_MOON_OP_TIMEOUT), but no reply ever arrived — the connection stayed open and the server simply never responded, so the timeout's failure path didn't engage as expected against a spinning (vs. cleanly-dead) server.

Impact

A single degenerate ingest payload can wedge a whole shard indefinitely. There
is no server-side ceiling that bounds how long one request can hold the shard
spinning.

Repro pointer

LongMemEval-S longmemeval_s offset 57 (qid 1faac195), ingested through
Lunaris ingest_episode (chunk + RAPTOR community tree + vector/keyword FT.*
upserts). A minimal FT.*/HSET-only repro that drops the Lunaris layer is
pending — will follow once isolated (the host is mid-benchmark).

Questions / asks

1. Any known FT.* / HNSW degenerate-input path that can spin (e.g. a
   pathological vector batch, an oversized field, or an index-rebuild loop)?
2. Could sustained memory pressure push a shard into an unbounded alloc/GC spin
   rather than failing fast?
3. Would a server-side per-command watchdog / CPU budget be feasible, so one
   request can't hold a shard spinning and unresponsive indefinitely?

[TinDang97]

Update — likely environmental (memory pressure), not data-dependent.

I re-ran the exact trigger question (qid 1faac195, offset 57) in isolation
against a freshly-started shard, on the same host but with memory pressure
eased and one-question-at-a-time isolation (FLUSHALL between questions). It
completed cleanly in ~9 min — no spin, no unresponsiveness.

So the original 40-min wedge does not reproduce from offset-57's payload
alone. The differentiator was the host state at wedge time: swap ~87% full,
~2.5 GB free, many concurrent ingests' worth of accumulated state. That points
at memory-pressure-induced thrashing (question 2 in the issue) rather than a
degenerate FT.*/HNSW input (question 1). Both the shard (~266% CPU) and the
client-side embedder (~192% CPU) appear to have been victims of the same
swap-thrashing, not a code-level infinite loop on specific data.

The robustness ask still stands and is arguably the more valuable one: a
server-side per-command watchdog / CPU budget (question 3) so a shard under
pressure fails fast / sheds load instead of spinning silently and unresponsive
for 40+ minutes with no checkpoint progress.

[pilotspacex-byte]

Closing — this is the SQ8/TQ code-size mis-dispatch CPU error-storm, fixed in v0.7.1.

Mechanism match

The repro fingerprint lines up exactly with the bug fixed in commit f5641707
(PR #334, "fix(vector): SQ8/TQ code-size mis-dispatch CPU error-storm"):

• auto_index_hset (src/shard/spsc_handler.rs) calls idx.try_compact() on
  every vector-bearing HSET (the "Insert-path compaction trigger" — same
  calls the FT.SEARCH path makes per query, but now on every write too).
• try_compact() → begin_background_merge_due() → needs_merge()
  (src/vector/store.rs), which — once an index has an immutable segment with

  20% dead entries (compaction::needs_vacuum) — computed
  live_count * self.collection.bytes_per_code_per_vector().

• For an SQ8-quantized index (SQ8's bits() == 8, outside TurboQuant's
  supported 1..=4 range), bytes_per_code_per_vector() fell through to the
  free code_bytes_per_vector(padded, bits) helper's _ => arm, which
  returned 0 and called tracing::error! on every single call (no
  once-only latch existed pre-fix).
• Because this ran synchronously on the shard's single event-loop thread and
  fired on every HSET once vacuum-eligible dead entries existed, sustained
  ingest with repeated upserts (Lunaris ingest_episode: HSET hydration docs
  • FT vector/keyword upserts, updating existing keys → dead entries) turned
    into a tight error-logging storm that pinned the CPU and starved the event
    loop — explaining the ~266% CPU spin, PING blocking indefinitely (listener
    up, event loop wedged), and no further persistence_tick progress (same
    starved loop). SQ8 is also the default/recommended quantization for
    ≤384d workloads (e.g. MiniLM embeddings), consistent with a RAG-style
    ingest pipeline like Lunaris.

Fix

f5641707 (PR #334) makes CollectionMetadata::code_bytes_per_vector() and
bytes_per_code_per_vector() dispatch SQ8 to its real layout (dim unpadded
u8 codes + 8-byte affine (min,scale) trailer) instead of falling through to
the TurboQuant-only helper, and adds an AtomicBool once-only latch to the
free helper's error log as defense in depth for any future stray caller.
Released in v0.7.1 (commit 4161cdc0).

Test evidence

Regression test added in the same commit, run on current main (4161cdc0,
macOS aarch64):

test vector::turbo_quant::collection::tests::test_code_bytes_per_vector_sq8 ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 4328 filtered out

Reopen condition

Reopen if this recurs on ≥v0.7.1 — that would indicate a different
degenerate-input path (the original issue's questions about a general
per-command CPU watchdog/budget are still open design questions worth tracking
separately, but the specific spin-and-wedge mechanism reported here is fixed).

Author: Tin Dang tindang.ht97@gmail.com