Skip to content

Audit whether #2616's merged drop-fencing protocol closes the handle-leak/race gaps #2470's review found in its own (now-superseded) drop protocol #2891

Closed

Description

@kriszyp

Background

PR #2470 (fix/integration-flake-triage) spent ~45 rounds of independent review building a fail-closed cross-worker database-drop protocol (sig 2 of that investigation): a reused .restoring lifecycle marker, an ITC close_database signal, a process-wide registryStatus() closure check, and a 409 refusal when a handle survives the broadcast — closing a RocksDB LOCK-held-forever wedge (open racing a destroy) and a long tail of handle-leak defects the review rounds found along the way.

While #2470 sat in review, origin/main independently merged its own solution to the same problem in PR #2616 ("Activate native full-text indexes with shared lifecycle coordination", merged 2026-09-26): resources/databaseDropPreparation.ts's claimDatabaseDropPreparation()/DatabaseDroppingError (409), per-root drop locks, and registryStatus()-based closure checks of its own — built for a different original purpose (full-text derived-index teardown) but covering the same destructive-DDL-vs-process-global-native-handle problem.

Per-maintainer decision, #2470 is being recut: sig 1 (already fixed on main), sig 3 (test fix), and sig 4 (Windows process-tree PID recycling) landed independently as #2890. Sig 2 — the drop protocol itself — is not being ported; #2616's protocol is the one going forward. This issue is the remaining piece.

The task

Audit origin/main's dropDatabase()/resumeIncompleteDatabaseDrop()/databaseDropPreparation.ts (from #2616) against the specific defects #2470's ~45 review rounds found and fixed in its own (now-abandoned) protocol, and port forward only what's actually missing. This is an audit-then-implement task, not a revival of #2470's marker-based architecture.

Confirmed already present in #2616 (read directly against source, not assumed):

  • Fail-closed 409 via registryStatus(), checked both before and after the destructive close (resources/databases.ts dropDatabase())
  • Crash recovery via resumeIncompleteDatabaseDrop() with a drop marker (databaseDropMarkerPresent)
  • Per-root drop locks and blob-path resolution at drop time

Not yet verified either way against #2616 — the specific gaps to check, each with the #2470 commit/round that fixed it as a starting point:

Why this matters (if gaps are real)

The original driving bug: an open racing a destroy left a RocksDB LOCK held forever after a crash — every subsequent database open on that path fails, job workers die at boot. Structural trigger (any concurrent drop + rescan/open under real load), unbounded (no self-recovery without operator intervention), and whatever fraction of the handle-leak tail isn't independently closed by #2616 reopens some slice of that same failure class.

Scope note

Not a blind port. #2470's marker-based architecture is superseded — the ask is specifically: does #2616 already close each gap above, and if not, what's the smallest patch on top of #2616's existing protocol (not a parallel one) to close it. Expect this to need the planning-review gate (prepush-review.mjs --mode plan) before implementation, since it's core shared database-lifecycle code.

Refs #2470, #2616, #2890.

Activity

  1. added theissue type on Sep 28, 2026
  2. self-assigned this
    on Oct 20, 2026
  3. kriszyp commented on Oct 23, 2026

    @kriszyp
    MemberAuthor

    Closing: the six-gap audit is done and Remove a dropped LMDB database's file only after its environment has closed (#3321) fixed the last open correctness gap. It merged to main on 2026-10-23 (v5.4).

    # Gap Outcome
    1 Index handle leak when a declaration throws Closed by #3201 (owned-store tracking)
    2 LMDB close before unlink Fixed by #3321: the drop removes the file, and a peer acknowledges it, only after the environment has closed
    3 Streamed directory removal Accepted as-is, not patched (see below)
    4 Marker replaced between two reads Closed by #2616 (separate drop marker) and #2649 (re-read under the lock)
    5 Table.cleanup() ordering Closed by #2616
    6 Marker records its own blob roots Closed by #2616

    Gap 3, accepted: fs.rm starts removing every file at once, which can occupy all four libuv threadpool slots. In a measurement, a concurrent stat stalled for 404 ms while a 25.6 GB tree (400 × 64 MB files) was deleted, against 0.2 ms with streamed removal. Harper only uses fs.rm for detached roots and crash recovery; normal drops go through native destroy() or unlink. The stall is bounded and recovers on its own, and the normal RocksDB drop path already blocks the owner thread for the same deletion. If it ever shows up in practice, the fix is to switch those two rm calls to a streamed loop.

    🤖 Posted by Claude Code (Claude Opus 5.5) on behalf of @kriszyp.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Fields

Priority

P1

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions