Skip to content

Make the replay-state and derived-index handoff tests wait for the real condition - #4309

Merged
kriszyp merged 2 commits into
mainfrom
fix/deflake-replay-state-and-index-handoff-tests
Nov 12, 2026
Merged

kriszyp merged 2 commits into
mainfrom
fix/deflake-replay-state-and-index-handoff-tests

Conversation

@kriszyp

@kriszyp kriszyp commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

⊙ Problem

Main's Unit Test went red at 9d77c42, with one failing test on each of two legs. Both are timing races in the tests. Neither comes from #3504, which only changed history reads.

  • replayCommitBoundaries copies txn.state right after flush(). rocksdb-js writes that file from a flush listener that can still be running then, so the copy found no file.
  • The derived-index handoff test applied the old owner's batch on a 50 ms timer and assumed the handoff would finish first. On a slow runner it did not, so the batch was applied.

💡 Solution

Each test now waits for the condition it depends on:

⚖️ Alternatives

  • Revert Keep a record's audit history readable after it is evicted #3504 — neither failure reaches its code, and its own CI was green.
  • Longer sleeps or a bigger applyDelay — still a timed race, only rarer, and slower.
  • Make rocksdb-js flush() wait for its listener — production never reads txn.state right after a flush, so only the test needs it.

🤔 Is the log's flushed position the right thing to wait on?

The wait polls getStats().lastFlushedPosition instead of decoding the file. In rocksdb-js 2.14.0 that value is read from txn.state itself, under the lock its writer holds, and a missing or short file reads as {0,0}. So it cannot run ahead of the file. Switching to decoding the bytes is a local test edit.

🤔 Does holding the batch narrow the handoff test?

The old owner's apply timer is now started after the epoch flips, not during the stalled shutdown. The fake checks the epoch when the timer fires, so the start time changes nothing it tests. The test still fails if fencing breaks.

✅ Verification

  • Handoff test, fails on base: with 40 ms added to the handoff, base fails with the CI assertion. The fix passes with 40 ms and with 480 ms added.
  • Replay child, fails on base under fault injection: hiding txn.state for 300 ms after the flush gives base the CI's ENOENT; the fix waits.
  • npm run test:unit:resources: 8067 passing, 2 failing in files this PR does not touch (details below).
Fault-injection cases and the two unrelated failures
  • Probes ran on throwaway copies of the crash child. With an older position written into txn.state for 300 ms, base saves that older position and the fix waits for the seed's. With the file never appearing, the fix fails after 10 s, naming the position it waited for.
  • rocksOpenConfig fails the same way on this Mac every run; that shard is green in CI. originClosedFloor passed on two reruns.
- Both changed files pass alone under RocksDB, LMDB and `--conditions=typestrip`; `npm run lint` and Prettier are clean. - End-to-end: not observable end-to-end (test-only change).

PR QA: skipped @ af8f0cc — test-only change

Docs: none needed (test-only change).

🤖 Generated by Anthropic Claude (Opus 5.5) in Claude Code; posted via @kriszyp.

🤖 Generated with Claude Code

Release target: v5.4 — test-only deflake; v5.3 has the same two tests if its CI starts flaking on them
Related PRs: #2532 overlaps (adds a weaker txn.state wait to the same crash child; whichever lands second should keep one helper with this PR's seed-position check)
Complexity: easy

Deferred findings

  • a8cf7d9ec9c77435 nit test-coverage — unitTests/resources/replayCommitBoundaries-crash.js — seed-state wait polls getStats witness, not the on-disk txn.state bytes it copies — NOT FILED
  • 574bef41391faa2d nit test-coverage — unitTests/resources/derivedIndexRuntimeNativeBackend.test.js — no-op scheduleApply drops the timer-set-during-stalled-shutdown window from the handoff test — NOT FILED

Accepted behavior

Behavior the owner ruled intended, so there is nothing to file.

1 minor or nit
  • 14379ce77ef5176c nit test-coverage — unitTests/resources/derivedIndexRuntimeNativeBackend.test.js — fixed sleep(20) before asserting the second owner still waits for the lock — accepted: repository owner's AGENTS.md 'Test timing' rule: fixed sleeps are reserved for asserting a non-event, which this sleep(20) does (the second owner still waits for the lock)
Origin — the dispatch brief this PR was written from

harper main is red: Unit Test at 9d77c42

main on HarperFast/harper went from green to red. Failing workflows: Unit Test at 9d77c42.

Find the merge that broke it and get main green. Walk that workflow's recent push runs on main oldest-to-newest to find the FIRST red head SHA and the PR it came from, then pull the failing test names from that run and from the current head. A failure on every matrix leg (Node version, Bun, uWS, Windows) is a regression; one leg is usually a flake.

Then either fix it or, when the culprit is a single merge and the fix is not obvious, open a revert and say so — main being green is worth more than the change being preserved. Record the fingerprint on the initiatives/ci-health board doc either way (prose row beginning with the test path, never a bare ref), and add your PR to initiatives/testing-and-deploy > "CI & build infrastructure" as a bare-ref line so it renders as a card.

Dispatch: task main-red-kriszyp_harper_9d77c4270_4d7eefda · queued by automation · ran by claude/opus/xhigh · worker kris-mac-m3-fleet-18b9c6

Review-Coverage: authored=claude; ran=cursor-composer,gemini,codex,gemini-carbon; late=gemini-carbon:adjudicated; adjudicated=domain; declined=cursor-grok,muse; rounds=3; full=2 @ af8f0cc

Review-Attention: skim ~2m (decisions: handoff-hold-mechanism) @ af8f0cc

kriszyp and others added 2 commits November 12, 2026 08:51
Two unit tests raced on timing and turned main red at 9d77c42, each on
one leg; neither involves that merge.

The replay crash child read transaction_logs/local/txn.state right after
flush(), but rocksdb-js writes it from a flush listener that can still be
running then, so the read could find no file (ENOENT) or an older
position. The child now waits until the log reports the seed's position
as flushed before it saves the state.

The derived-index handoff test let the old owner's batch apply on a 50 ms
timer and assumed the handoff finished first; on a slow runner the apply
ran while that owner still held the epoch. The test now holds the batch
until the epoch flips, then applies it and checks it was fenced.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: main-red-kriszyp_harper_9d77c4270_4d7eefda
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: main-red-kriszyp_harper_9d77c4270_4d7eefda
@kriszyp kriszyp added this to the v5.4 milestone Nov 12, 2026
@kriszyp
kriszyp marked this pull request as ready for review November 12, 2026 18:50
@kriszyp
kriszyp merged commit 608fcbd into main Nov 12, 2026
67 of 70 checks passed
@kriszyp
kriszyp deleted the fix/deflake-replay-state-and-index-handoff-tests branch November 12, 2026 18:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant