Skip to content

Serve a stopped delegate's keys at once after a fenced record-lock home-map change - #3269

Merged
kriszyp merged 14 commits into
mainfrom
feat/record-lock-generation-expires-old-grants
Oct 20, 2026
Merged

kriszyp merged 14 commits into
mainfrom
feat/record-lock-generation-expires-old-grants

Conversation

@kriszyp

@kriszyp kriszyp commented Oct 18, 2026 •

Copy link
Copy Markdown
Member

⊙ Problem

After a record-lock home map moves to a new generation, a home keeps its grants from the old one. If that delegate was stopped for good, the home answers "contended" for the key for about 365 s. harper-pro's fenced transition (harper-pro#973) hits this on every key the stopped node held from a surviving home.

🤔 Should core change, rather than harper-pro waiting the lease out?

Yes: without it, a fenced transition activates in seconds but serves those keys six minutes later. The maintainer approved this core scope. Only grants the drain already ruled on are retired, so reverting restores today's wait.

💡 Solution

  • A home serves the keys a stopped node held at once after a fenced activation. It retires only older grants the stage drain reported outstanding on a node the new ring drops.
  • Every other older grant keeps its lease, as the backstop against an early activation.
  • A drain no longer reports "complete" while authority is still live (harper-pro#1061).

⚖️ Alternatives

  • Expire every older grant — loses the lease backstop for delegates still in the ring.
  • Infer "the drain ruled" from recall history — a contention recall looks the same.
  • Also retire recall-confirmed grants — a recall can confirm while a renewal's admission still runs.

🤔 May core retire a dropped node's drain-marked grant without knowing it was fenced?

Yes, by design. harper-pro activates past an unconfirmed recall only on a stop attestation or after the lease wait. Activating early is the same operator error as fencing a running node. Passing fences into core would be a new interface.

🔧 Changes

✅ Verification

  • npx mocha unitTests/resources/recordLock*.test.js: 326 passing, with new tests in unitTests/resources/recordLockCoordinator.test.js under "a home across a generation change" and quiesceDelegations.
  • Removing any one of these fails its new test: the retire rule, the re-arm, the re-read loop, the surrender join, or the cold-close ownership check.
  • End to end: the harper-pro PR's recordLockApplyFenced.test.mjs kills a node and fences it. It then re-locks that node's exact keys at once; without this change, those re-locks fail.
New tests

Under "a home across a generation change":

  • dropped and drained;
  • unchanged generation;
  • in ring;
  • recall-confirmed in ring;
  • recall still in flight at the drain deadline;
  • never drained after a failed contention recall;
  • a recall during an in-flight surrender;
  • a later drain re-arming the sweep;
  • a coordinator rebuilt while the drain waits, then cold-closed;
  • a late release that must not clear its successor.

Under quiesceDelegations: a coordinator rebuilt mid-sweep.

🤖 Generated by Claude Code (Claude Opus); posted via @kriszyp.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Kcpk75VwseBHAGwqT6mQvP

Release target: v5.4 — the core half of a new harper-pro feature (harper-pro#973), shipping with v5.4; no backport
Related PRs: #3248 overlaps (same coordinator test file; successors on the recovery path now take no lineage), #3259 independent (this branch is based on it)
Complexity: complicated

Deferred findings

  • 5f78fffa79c8bea3 minor concurrency — resources/recordLockCoordinator.ts — mismatched-token recall confirms while the older-token delegation admits; drain counts it recalled — NOT FILED
  • 401782fcd8a36fd7 minor reliability — resources/recordLockCoordinator.ts — drain in recall backoff marks the grant outstanding without sending a recall — NOT FILED

Review-Coverage: authored=claude; ran=gemini,muse,codex; adjudicated=domain; declined=cursor-grok,cursor-composer,cursor-kimi,cursor-muse; rounds=13; full=7 @ afc0cd7

Review-Attention: deep ~25m (critical: recordLockCoordinator.ts; decisions: retire-on-inferred-fence, handed-off-ownership-reports-zero, do-less-alternative) @ afc0cd7

@kriszyp kriszyp added this to the v5.4 milestone Oct 18, 2026
gemini-code-assist[bot]

This comment was marked as resolved.

kriszyp and others added 14 commits October 19, 2026 07:53
… current

A home kept a grant minted under generation g after g+1 activated, and answered
contended on that key until the delegation lease ran out (~365 s). When the
delegate is a node stopped outside Harper (harper-pro#973's fenced transition),
that is a key nobody can admit under. The first grant under a new generation now
expires every older grant, the home-side half of "a delegation is authority
within one generation".

Dispatch-Task: hp-973-fenced-apply-homes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kcpk75VwseBHAGwqT6mQvP
Expire an older-generation grant only when its delegate is outside the new
homes[] or already confirmed its recall. A delegate the new generation still
names keeps its lease, the backstop against an activation that came too early.
Document that a drain proof or a stopped (not merely isolated) fenced node is
what lets activation skip the lease-plus-skew wait.

Dispatch-Task: hp-973-fenced-apply-homes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kcpk75VwseBHAGwqT6mQvP
Dispatch-Task: hp-973-fenced-apply-homes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kcpk75VwseBHAGwqT6mQvP
A drain that never reached a grant (for example one moved to a successor
coordinator mid-sweep) must not let the home re-grant the key at activation.
Expire an older grant only when the delegate confirmed its recall, or when the
recall failed and the delegate is outside the new homes[]; harper-pro activates
past such a recall only when the operator attests the node stopped. Every other
older grant keeps its lease. Isolation is not a fence for skipping the wait.

Dispatch-Task: hp-973-fenced-apply-homes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kcpk75VwseBHAGwqT6mQvP
Recall history cannot tell a drain's recall from a contention recall, and a
drain recall still in flight at the drain deadline has no failure recorded when
the first g+1 grant sweeps. quiesce now marks each grant it reports
outstanding; the sweep expires an older grant only when its recall was
confirmed, or the drain marked it and its delegate is outside the new ring.

Dispatch-Task: hp-973-fenced-apply-homes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kcpk75VwseBHAGwqT6mQvP
Dispatch-Task: hp-973-fenced-apply-homes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kcpk75VwseBHAGwqT6mQvP
…wed grant's drain mark

A recall that confirms after the first grant under the new generation was left
answering contended until its lease when the release was lost; the next grant
request now expires it. A renewal re-mints the token, so the drain mark that
described the old token goes with it.

Dispatch-Task: hp-973-fenced-apply-homes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kcpk75VwseBHAGwqT6mQvP
…mark when a drain reaches it

quiesceDelegations swept a snapshot of the database's coordinators. Staging a
generation replaces the transport, and the table's next lock access rebuilds
its coordinator, adopting the grants and delegations; a predecessor the sweep
had not reached yet returned at once as closed, and the successor was never
swept, so the drain could report complete with authority still live
(harper-pro#1061). Re-read the live coordinators after each pass until none is
new. A drain also clears the mark an earlier drain left before recalling, so
the mark always describes the latest drain that reached the grant.

Dispatch-Task: hp-973-fenced-apply-homes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kcpk75VwseBHAGwqT6mQvP
…it for surrenders in a drain

A surrender removes the delegation before its handles are fenced; a recall
arriving in that window found nothing and confirmed, so the home could treat a
grant as drained (and, under the generation sweep, retire it) while a handle
could still commit. Such a recall now joins the in-flight surrender, and
quiesceDelegations waits for in-flight surrenders and reports one that does not
settle. The sweep and the per-key late branch share one predicate, and expiring
under a generation records that generation as acted on, so a map that reverts
below it cannot re-grant the key. A coordinator closed mid-drain stops, leaving
its successor to the next pass, and every visited coordinator's ownership
horizon is checked again.

Dispatch-Task: hp-973-fenced-apply-homes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kcpk75VwseBHAGwqT6mQvP
…mint, and count a drain once

- The generation sweep runs at the first mint under a generation rather than on
  any map read, so a read that mints nothing neither expires a grant nor raises
  the floor.
- A recall-confirmed grant is no longer retired: a recall can confirm early
  while a renewal's inherited admission still runs, and its release normally
  follows the confirmation anyway.
- A sweep that visits a successor counts each delegation, grant and in-flight
  surrender once, and a handed-off coordinator reports no ownership gap of its
  own.

Dispatch-Task: hp-973-fenced-apply-homes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kcpk75VwseBHAGwqT6mQvP
…ty, and raise the floor on retiring

- The drain waits on in-flight surrenders inside the re-read loop, so a
  coordinator rebuilt during that wait is swept and its adopted ownership
  checked; a handed-off predecessor no longer answers for it.
- The generation sweep runs once per generation from a grant request that
  passed the quarantine and fencing gates, before the capacity checks, so grants
  to a dropped node do not hold the table at capacity.
- Retiring a grant records its generation as acted on, so a map that later
  reads below it cannot re-grant the key.
- A drain clears a grant's mark before counting a confirmed recall, and the
  retire predicate excludes confirmed recalls.

Dispatch-Task: hp-973-fenced-apply-homes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kcpk75VwseBHAGwqT6mQvP
…losed successor's ownership

A drain that marks a grant after this generation's sweep re-arms it, so the
next grant request retires every marked grant, not only the requested key's.
A handed-off coordinator whose successor closed without handing on reports the
successor's ownership gap instead of none.

Dispatch-Task: hp-973-fenced-apply-homes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kcpk75VwseBHAGwqT6mQvP
…the lease is no backstop

A drain that marks a grant re-arms the generation sweep on the end of the
handoff chain, not on a predecessor it was handed off from mid-drain. The design
note now says the lease backs up an early activation only for grants a drain did
not mark: activating past a drain-marked grant to a dropped node is the
operator's statement that the node is stopped.

Dispatch-Task: hp-973-fenced-apply-homes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kcpk75VwseBHAGwqT6mQvP
Dispatch-Task: hp-973-fenced-apply-homes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kcpk75VwseBHAGwqT6mQvP
@kriszyp
kriszyp force-pushed the feat/record-lock-generation-expires-old-grants branch from 6b45e61 to afc0cd7 Compare October 19, 2026 14:13
@kriszyp
kriszyp merged commit 1fd591b into main Oct 20, 2026
52 of 57 checks passed
@kriszyp
kriszyp deleted the feat/record-lock-generation-expires-old-grants branch October 20, 2026 02:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant