Repository navigation
Make the replay-state and derived-index handoff tests wait for the real condition - #4309
Merged
Merged
Conversation
Two unit tests raced on timing and turned main red at 9d77c42, each on one leg; neither involves that merge. The replay crash child read transaction_logs/local/txn.state right after flush(), but rocksdb-js writes it from a flush listener that can still be running then, so the read could find no file (ENOENT) or an older position. The child now waits until the log reports the seed's position as flushed before it saves the state. The derived-index handoff test let the old owner's batch apply on a 50 ms timer and assumed the handoff finished first; on a slow runner the apply ran while that owner still held the epoch. The test now holds the batch until the epoch flips, then applies it and checks it was fenced. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Dispatch-Task: main-red-kriszyp_harper_9d77c4270_4d7eefda
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Dispatch-Task: main-red-kriszyp_harper_9d77c4270_4d7eefda
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
⊙ Problem
Main's Unit Test went red at 9d77c42, with one failing test on each of two legs. Both are timing races in the tests. Neither comes from #3504, which only changed history reads.
replayCommitBoundariescopiestxn.stateright afterflush(). rocksdb-js writes that file from a flush listener that can still be running then, so the copy found no file.💡 Solution
Each test now waits for the condition it depends on:
txn.state, in every mode that saves it.⚖️ Alternatives
applyDelay— still a timed race, only rarer, and slower.flush()wait for its listener — production never readstxn.stateright after a flush, so only the test needs it.🤔 Is the log's flushed position the right thing to wait on?
The wait polls
getStats().lastFlushedPositioninstead of decoding the file. In rocksdb-js 2.14.0 that value is read fromtxn.stateitself, under the lock its writer holds, and a missing or short file reads as{0,0}. So it cannot run ahead of the file. Switching to decoding the bytes is a local test edit.🤔 Does holding the batch narrow the handoff test?
The old owner's apply timer is now started after the epoch flips, not during the stalled shutdown. The fake checks the epoch when the timer fires, so the start time changes nothing it tests. The test still fails if fencing breaks.
✅ Verification
txn.statefor 300 ms after the flush gives base the CI's ENOENT; the fix waits.npm run test:unit:resources: 8067 passing, 2 failing in files this PR does not touch (details below).Fault-injection cases and the two unrelated failures
txn.statefor 300 ms, base saves that older position and the fix waits for the seed's. With the file never appearing, the fix fails after 10 s, naming the position it waited for.rocksOpenConfigfails the same way on this Mac every run; that shard is green in CI.originClosedFloorpassed on two reruns.PR QA: skipped @ af8f0cc — test-only change
Docs: none needed (test-only change).
🤖 Generated by Anthropic Claude (Opus 5.5) in Claude Code; posted via @kriszyp.
🤖 Generated with Claude Code
Release target: v5.4 — test-only deflake; v5.3 has the same two tests if its CI starts flaking on them
Related PRs: #2532 overlaps (adds a weaker txn.state wait to the same crash child; whichever lands second should keep one helper with this PR's seed-position check)
Complexity: easy
Deferred findings
a8cf7d9ec9c77435nit test-coverage —unitTests/resources/replayCommitBoundaries-crash.js— seed-state wait polls getStats witness, not the on-disk txn.state bytes it copies — NOT FILED574bef41391faa2dnit test-coverage —unitTests/resources/derivedIndexRuntimeNativeBackend.test.js— no-op scheduleApply drops the timer-set-during-stalled-shutdown window from the handoff test — NOT FILEDAccepted behavior
Behavior the owner ruled intended, so there is nothing to file.
1 minor or nit
14379ce77ef5176cnit test-coverage —unitTests/resources/derivedIndexRuntimeNativeBackend.test.js— fixed sleep(20) before asserting the second owner still waits for the lock — accepted: repository owner's AGENTS.md 'Test timing' rule: fixed sleeps are reserved for asserting a non-event, which this sleep(20) does (the second owner still waits for the lock)Origin — the dispatch brief this PR was written from
harper main is red: Unit Test at 9d77c42
mainon HarperFast/harper went from green to red. Failing workflows: Unit Test at 9d77c42.Find the merge that broke it and get main green. Walk that workflow's recent push runs on
mainoldest-to-newest to find the FIRST red head SHA and the PR it came from, then pull the failing test names from that run and from the current head. A failure on every matrix leg (Node version, Bun, uWS, Windows) is a regression; one leg is usually a flake.Then either fix it or, when the culprit is a single merge and the fix is not obvious, open a revert and say so — main being green is worth more than the change being preserved. Record the fingerprint on the
initiatives/ci-healthboard doc either way (prose row beginning with the test path, never a bare ref), and add your PR toinitiatives/testing-and-deploy> "CI & build infrastructure" as a bare-ref line so it renders as a card.Dispatch: task
main-red-kriszyp_harper_9d77c4270_4d7eefda· queued by automation · ran by claude/opus/xhigh · worker kris-mac-m3-fleet-18b9c6Review-Coverage: authored=claude; ran=cursor-composer,gemini,codex,gemini-carbon; late=gemini-carbon:adjudicated; adjudicated=domain; declined=cursor-grok,muse; rounds=3; full=2 @ af8f0cc
Review-Attention: skim ~2m (decisions: handoff-hold-mechanism) @ af8f0cc