Flaky: 'Unit tests: lmdb' SIGSEGV (exit 139) in subscriptionReplay edge cases, around '!omitCurrent: empty primaryStore delivers nothing initially' — Node 22, 24 and 26 #3349

Activity

1. Ethan-Arrowood

   added

   bugSomething isn't workingSomething isn't working

   flaky-testTest known to fail intermittently, unrelated to the PR's own diffTest known to fail intermittently, unrelated to the PR's own diff

   on Oct 20, 2026
2. Ethan-Arrowood

   added a parent issue on Oct 20, 2026

   • [Epic] CI test flakiness #1655
3. Ethan-Arrowood

   added the

   Bug

   issue type on Oct 20, 2026
4. Ethan-Arrowood

   set

   Priority

   to

   P2

   on Oct 20, 2026
5. kylebernhardy commented on Oct 31, 2026

   

   kylebernhardy

   on Oct 31, 2026

   Member

   More actions

   Observed another Node 26 occurrence in Adopt fulltext 0.8 with host query policy and fuzzy controls, at 8bf60a4: failed LMDB job.

   • The repaired encoder worker test passed in 995 ms. The later failure was a native segmentation fault, exit 139; no Mocha assertion or epilogue reported.
   • At 01:13:56.224 UTC, native LMDB printed:

     cursor_close: mc->mc_txn or mc->mc_txn->mt_env is NULL, ensure that cursors are closed before the transaction is closed
     

   • The deleted-key and nonexistent-record subscription cases then passed. The last pass was at 01:13:56.710; the process crashed at 01:14:08.627, approximately 12 seconds later. The next test was non-collection: record with single version (no version chain) delivers just current.
   • Verified at the failed head: the subscription replay fixture and transaction broadcaster are unchanged against both the PR base and current main. This occurrence extends the previously recorded edge-case window by one test.

   The location and timing resemble the existing reports. The warning adds evidence of an invalid cursor/transaction lifetime, but neither it nor matching timing establishes that every occurrence has the same native fault.

   A separate fixture ownership defect is verified in the cleanup patch: 35 calls used subscription.return?.(), but the subscription queue exposes end()/close(); return() belongs to its iterator. collect() attaches data listeners, so it does not implicitly close an iterator. A real-table check asserting closed/unregistered state failed before and passed after correcting cleanup. The fixture now tracks subscriptions from all 43 creation sites and ends them after each test, including assertion-failure paths. The actual iterator-return case remains, with stronger lifecycle/no-late-delivery assertions.

   Cleanup verification: full LMDB fixture passed 43 cases each on Node 24/26; RocksDB passed 34 with 9 existing skips. Final focused iterator/idle checks also passed. These checks prove fixture cleanup, not the native SIGSEGV cause or resolution. This issue remains open.

   Next diagnostic: capture an all-thread native backtrace at SIGSEGV while preserving the existing LMDB workload and assertions, for example by running the affected Mocha process under a native debugger. A Node fatal-error report alone is insufficient evidence that an operating-system SIGSEGV will be captured. Keep raw cores private; do not add speculative cursor fixes or treat a passing rerun as resolution. No CI job was rerun.

   Comment generated by kAIle (GPT-6)
6. kriszyp

   self-assigned this

   on Nov 2, 2026
7. kriszyp commented on Nov 3, 2026

   

   kriszyp

   on Nov 3, 2026

   Member

   More actions

   Root cause found. The defect is in lmdb-js 3.5.9, the newest release; a fix is ready but not yet pushed.

   Native stack. Print a native backtrace when a unit-test step segfaults in CI keeps cores in the unit-test job. Its own lmdb leg then crashed in outOfOrderWriteExpiry.test.js and printed the stack (log):

   Thread 1 (main), SIGSEGV
   #0 mdb_cursor_close (mc=...) at mdb.c:10432      MDB_CURSOR_UNREF reads mc->mc_txn->mt_env->me_flags
   #1 CursorWrap::close (...) at src/cursor.cpp:60   JS cursor.close()
   

   The faulting load is the me_flags read through a freed MDB_txn. When the freed memory happens to be zero, the same call prints cursor_close: mc->mc_txn or mc->mc_txn->mt_env is NULL instead, the warning seen on 10-31.

   The 9–14 s silence before each crash came after the fault, not before it. It is the time spent writing the core: about 1 GB, piped to apport on the runner. With cores written to disk, the gap is 4 s.

   Violated invariant. finishCursor (lmdb-js read.js) parks a finished read cursor in db.availableCursor, bound to the current read txn. A later iterator renews it only when it first positions. Two paths free that txn while the cursor is still parked:

   • resetReadTxn orphans it, and Txn.done or releaseToRenewingCursors then aborts it.
   • _commitReadTxn, which runs on every openDB (every Harper table creation), commits it.

   An iterator that takes the parked cursor and is closed before it positions then calls mdb_cursor_close on freed memory. releaseToRenewingCursors does exactly that to a snapshot: false iterator suspended across a tick.

   Deterministic reproduction with plain lmdb 3.5.9, under valgrind:

   Invalid read of size 8
      at mdb_cursor_close (mdb.c:10432)
      by CursorWrap::close (cursor.cpp:60)
    Address … is 72 bytes inside a block of size 870 free'd
      by mdb_txn_end (mdb.c:4092)
   

   Fix (lmdb-js, local branch, not pushed): keep a parked cursor only while its txn is current. Close the parked cursors before the read txn stops being current (both resetReadTxn orphan branches and _commitReadTxn) and before close() aborts it.

   Check	Base 3.5.9	Fixed
   valgrind on the reproduction	2 invalid reads	0 errors
   3 new lmdb-js tests asserting no cursor stays parked on a finished txn	fail	pass
   lmdb-js test/index.test.js	—	375 pass; 2 cross-env deadlock tests fail the same way on base
   Harper HARPER_STORAGE_ENGINE=lmdb full unitTests/resources	—	4872 pass, 0 fail

   Next: an lmdb-js PR and release, then a Harper bump of lmdb.

   _{Comment by KrAIs (Claude Opus 5.5)}
8. kriszyp

   added 3 commits that reference this issue on Nov 5, 2026

   Keep native crash cores in unit-test CI and print their backtraces on…

   ...

   779c002

   Close a parked cursor while its read transaction is still alive (#386)

   ...

   478288f

   Print a native backtrace when a unit-test step segfaults in CI (#3973)

   ...

   480655c