Skip to content

restore_backup: a stale pin can vouch for another restore's marker and purge an unclaimed target #3106

Closed

Description

@cb1kenobi

restore_backup's target_database admission treats the source repository's pin as proof that a restoring marker on the target is its own debris. A pin can outlive its marker, so a stale pin can vouch for a marker a different repository's restore wrote — and the restore then purges a target it never claimed, bypassing the documented non-destructive guarantee.

Introduced by #2649, which added the exemption so an interrupted restore could be rerun instead of wedging. The exemption itself is right; the ownership proof is too weak.

Why the pin is not proof

The marker body is the target directory name plus a timestamp (publishRestoringMarker, dataLayer/restoreMarker.ts:274-280). It records nothing about which repository the restore came from, so admission infers ownership from the presence of this source's pin:

const ourInterruptedRestore = preexisting && readBackupPins(backupDir).some((pin) => pin.pin_id === pinId);

dataLayer/rocksdbBackup.ts:1285. The pin id is derived from the target directory, so every source's pin for a given target has the same id — only which repository holds it differs.

That inference holds only while a pin never outlives its marker. It can:

  • releaseRestoreClaim (dataLayer/rocksdbBackup.ts:744-758) clears the marker first, then unpins. The unpin is wrapped in its own try/catch that logs and continues, so a failing unpin leaves the pin behind by design.
  • A crash between the two does the same.

Both are deliberate — the ordering exists so a crash never leaves an unloadable database whose source a purge may delete — and the stranded pin is documented as harmlessly lapsing, because pinIsLive ignores a pin whose marker is gone. That reasoning is sound for purge protection. It does not hold for the new ownership check, which asks a different question of the same artifact.

Sequence

  1. A restore from repository A into target T completes. It clears T's marker, then fails or crashes before unpinning. A's repository keeps a pin for T.
  2. A restore from repository B into T starts and is interrupted. T is left populated and marked.
  3. A restore from A into T is admitted: preexisting is true (B's marker), and A's stale pin matches. T is purged and replaced with A's data.

Step 3 is exactly the case target_database promises to refuse.

Impact

Bounded, and silent. T is already unloadable and awaiting a rerun that would purge it anyway, so no healthy database is destroyed — but B's operator loses the directory their rerun was going to recover, and nothing reports it. The restore succeeds normally.

The trigger needs all three steps, so this is unlikely to be hit in the field; it is filed because the invariant is wrong rather than because it is expected.

Fix

Record the source repository in the marker body and match on that instead of on the pin. markerIsIntact already treats line 1 as the contract (dataLayer/restoreMarker.ts), so a third line is additive.

A marker written by a version that does not record the source has no line to match. The upgrade path needs a decision: fall back to today's pin check for those (preserves the rerun fix, keeps the hole for pre-upgrade markers), or refuse them (closes it, at the cost of wedging a restore interrupted across the upgrade). Settling that is most of the work here.

Both the code comment at dataLayer/rocksdbBackup.ts:1283-1284 and the pin-lifetime note in dataLayer/DESIGN.md point at the epic for this gap; they should point here once this exists.

Not covered by tests

#2649 has no regression guard for this sequence — it would need a pin deliberately stranded past its marker, then a second source's marker on the same target.

Activity

  1. added
    bugSomething isn't working
    area:storageStorage engine, LMDB/RocksDB, compaction
    on Oct 8, 2026
  2. added theissue type on Oct 8, 2026
  3. cb1kenobi commented on Nov 14, 2026

    @cb1kenobi
    MemberAuthor

    Fixed by #3920 Publish every restore_backup through one crash-safe protocol, merged 2026-11-05 for v5.4. A pin only vouches for the restore that owns it, so a stale pin from another restore can no longer vouch for a target marker it never owned.

    🤖 Generated by Claude Opus 5.5; posted via @cb1kenobi.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:storageStorage engine, LMDB/RocksDB, compactionbugSomething isn't working

    Type

    Fields

    Priority

    P2

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions