[kriszyp]

⊙ Problem

A non-owner on a sharded table keeps each replica it reads until its deadline, however full the disk gets. The reclamation handler only acted on caching tables, and nothing reported how many replicas a node held.

Closes #4101
Closes #4105

🤔 Is L/p the right window?

The issue's window of L/p from now shrinks as pressure rises and is empty on a full disk. I used a lifetime of L/p, so remaining <= L - L/p.

🤔 Why v5.4 and not a backport?

The evidence points at v5.3, but local replicas are new in 5.4.

💡 Solution

• Below the free-space threshold, reclamation demotes replicas to stubs, oldest deadline first, down to a lifetime of replicaExpiration / priority.
• A record written in the last 30 seconds stays at any pressure, because its owner may not have it yet. A replicaExpiration of 30 s or less is not shortened.
• Owned rows and stubs are untouched. A later read fetches a demoted record again.
• describe_table reports replica_count and replica_size_estimate.

🤔 Is a row's version a good enough age?

The floor uses version as write time. A fetched old record can be demoted sooner, costing only a refetch. A backdated write can too, tracked in harper#4106.

🤔 Is demotion enough without compaction?

Demotion frees no disk until compaction rewrites the old rows. Compacting after a pass rewrites owned rows too and needs temporary space, so it needs its own design. The docs say so.

Disk on a raw store: 200k rows of 2.7 KB, half overwritten with short values

Step	Disk
Before overwrite	540 MB
After overwrite, flush and 30 s idle	541 MB
After compact()	472 MB (rows of 2 KB or more live in blob files)
After compact({ bottommost: true }), 2.5 s	272 MB

Rows under 2 KB: 85 MB, then 85 MB after the overwrite, then 45 MB after compact(). No runtime code calls compact() today, and caching-table eviction has the same limit.

⚖️ Alternatives

🤔 Is oldest-first worth its scans?

Oldest-first picks what survives an early stop, costing about 25% of a pass at 1M replicas. Moving the sweep's cutoff is smaller, but has no order or budget.

🤔 Should a pass hold up the path's other handlers?

A pass delays audit cleanup on that path. A run starts no batch after 60 s, but the limit never cuts one short.

Options ruled out

• [deadline, id] index — paid on every admission and renewal, with a backfill.
• One batch per run — at the hourly default, slower than the normal sweep.

Pass cost on a real table: 1M replicas of 1 KB, with owned-row writes on the same thread

bench-run (replica-reclaim-1m-20261106-121545), turbo off, Node 26.2, 20k batch, 60 s budget. Taken before the 30 s floor, which only changes which replicas are eligible.

Run	Entries	Demoted	Time	Rate	Event-loop delay p99 / max	Owned write p50 / p99
Priority 1.25 (includes the one-time repair walk)	1,000,000 to 794,285	205,715	56.3 s	3,654/s	12.4 / 307 ms	0.22 / 12.4 ms
Priority 2	794,285 to 480,898	313,387	56.4 s	5,557/s	1.5 / 357 ms	not recorded
Priority 2 again	480,898 to 464,236	16,662	3.6 s	4,589/s	1.9 / 263 ms	0.15 / 1.6 ms
Full volume, stopped by the budget	464,236 to 61,409	402,827	60.3 s	6,681/s	1.8 / 327 ms	0.15 / 1.7 ms
Full volume	61,409 to 0	61,409	8.5 s	7,237/s	2.7 / 57 ms	0.14 / 1.6 ms
Empty store	0	0	0.3 s	n/a	2.0 / 95 ms	0.49 / 10 ms

The maximum delays are garbage collection, inflated here by the benchmark's own array of 1M ids. A 500k run that released that array (replica-reclaim-gc) saw GC pauses of at most 84 to 96 ms and delays of at most 92 to 105 ms.

Selecting the earliest entries, on a raw store (replica-scan-20261106-114728):

Entries	Plain scan	Earliest 10k	Earliest 100k
100k	95 ms	105 ms	120 ms
1M	800 ms	860 ms	1110 ms
5M	4.0 s	4.17 s	4.6 s

🔧 Changes

• resources/Table.ts — ⚠️ Look hardest: the pass and its bounded scan. Also: the limits; the handler's guards; the pass as a sweep mode sharing the sweep's demotion; a late sweep owner registers the handler; getRecordCount on any store; getReplicaStats; clear() clears deadlines.
• The integration fixture is its config, schema and stand-in source.
• server/storageReclamation.ts — registration reports whether it registered, and getStorageReclamationPriority is shared with the pass.
• dataLayer/schemaDescribe.ts — the fields are linked in Solution, inside the existing skip_record_count gate.
• docs/reference/database/storage-tuning.md — the new section, including the 30 s floor and the compaction note.
• docs/reference/operations-api/operations.md, docs/reference/database/schema.md, docs/reference/configuration/options.md and the release note describe the same behavior.
• resources/DESIGN.md records why the window and budget bound a pass, and resources/MAP.md points at it.

What happens when disk runs low, and what an operator sees

Product and architecture tour

What happens when disk runs low

Before	After
A sharded table ignored storage pressure, and each replica lived to its deadline.	Each reclamation check demotes replicas older than replicaExpiration / priority, oldest deadline first.

Only a row flagged REPLICA is ever demoted, through the normal sweep's version-guarded eviction, so a replica renewed or overwritten since selection is kept.

One reclamation run on a table

1. Sample — Reads the priority fresh, since an earlier handler may have freed space, and stops at 1 or below.
2. Set the window — Takes lifetime = max(L / priority, min(L, 30 s)). The cutoff now + L - lifetime stays fixed, so new admissions cannot extend the run.
3. Select — Scans the deadline store for the earliest 20k at or before the cutoff, or beyond the current lifetime.
4. Demote — Skips a replica written under 30 s ago. Otherwise evicts with the observed deadline, which refuses a renewal or rewrite.
5. Re-check — Every 1000 attempts, stops if the priority is back to 1 or below.
6. Repeat or stop — Stops when all matches fit the batch, nothing moved, or 60 s have passed.

What an operator sees

describe_table adds these fields, counted the same time-limited way as record_count. They are absent where a table never kept a replica, on LMDB, and under skip_record_count. The count comes from deadline entries, and the size is sampled from up to 100 rows, so both are estimates on a large table.

✅ Verification

• Unit: the new suite passes on RocksDB; its acceptance cases fail on base. The capped-deadline case fails without the write-time check.
• End to end: a new integration test, reports a full quota on real workers: replicas become stubs, an owned row stays, a later read refetches.
• Not tested: the real Replicator across two nodes.

PR QA: Does a replica pressure pass stay correct while several HTTP workers read and write the same keys? — passed @ 8bcf674, qa-pr-harper-4107-3deb710

Full gates: CI on this head, and locally before the 30 s floor

CI on this head: every job passes, after reruns. lmdb on Node 26 took two: first ttlCleanupInterval, which also fails on main's lmdb job, then the 9-minute step limit. resources-indexes on Node 24 took one (derivedIndexRuntimeRocks, passes alone 3 of 3). Main's lmdb jobs were red on its last 8 runs.

Local gate	Result
test:unit:resources	7,284 passing, 1 failing: originClosedFloor, which passes alone 5 of 5 and also fails in other recent PR runs
test:unit:resources:core:o-z, on this head	3,491 passing
test:unit:main	10,569 passing, 2 failing: effective installation destination protection, also on base
test:integration:all	3,622 passing, 4 failing: all ollama-backend, no local models

Docs are in this PR. No Fabric follow-up: no new configuration.

_{🤖 Generated by Claude Opus 5.5; posted via @kriszyp.}

🤖 Generated with Claude Code

https://claude.ai/code/session_017xoBA8qucgMM6UHpQt3kZc

Release target: v5.4 — local replicas are new in 5.4, and the issue is milestoned there
Related PRs: #2155 overlaps (eviction scheduling in Table.ts), #3504 overlaps (what a demoted row keeps of its history), 3 others independent
Complexity: complicated

Deferred findings

• a7f4c4e1baa78f68 minor correctness — resources/Table.ts — Repaired replicas get a fresh lifetime, so the following pass cannot demote them at finite priority — filed Known limits of the replica storage-pressure pass #4106
• 631c3949331fd4cd minor correctness — resources/Table.ts — Under pressure a non-owner's fresh write replica can be demoted before its owner has the write — filed Known limits of the replica storage-pressure pass #4106
• 9c66c344117a55fa minor correctness — resources/Table.ts — replica_count counts deadline entries, not REPLICA rows; orphans count, deadline-less replicas undercount — filed Known limits of the replica storage-pressure pass #4106
• 22df7f484d218377 minor correctness — resources/Table.ts — Fresh-write floor uses record version; backdated context.timestamp on a TTL-capped write replica bypasses it — filed Known limits of the replica storage-pressure pass #4106
• 72ab9d5e5b7e6ea8 nit reliability — resources/Table.ts — A batch with no successful demotion ends the run — filed Known limits of the replica storage-pressure pass #4106
• 218edefe6cbda3f4 nit maintainability — integrationTests/server/replica-storage-reclamation.test.ts — Added comments narrate code (integration header, schemaDescribe parenthetical) — filed Known limits of the replica storage-pressure pass #4106

Origin — the dispatch brief this PR was written from

Reclaim sharded-table replicas under storage pressure

_{Dispatch: task harper-4101-replica-storage-reclamation · queued by unknown · ran by claude/opus/xhigh · worker kzyp-xps-1}

_{Ready-For-Promotion: 8bcf674 urgency=normal late=folded}

_{Review-Coverage: authored=claude; ran=cursor-composer,gemini,codex,cursor-grok; late=cursor-grok:adjudicated; adjudicated=domain; declined=muse; rounds=14; full=5 @ 8bcf674}

_{Review-Attention: deep ~35m (critical: schemaDescribe.ts, Table.ts +1; decisions: do-less-alternative, pressure-lifetime-divisor, fresh-write-floor-30s, sequential-awaited-pass, zero-expiration-exempt, describe-replica-fields, replica-count-from-deadlines, fixed-min-age, stats-gated-by-skip-record-count, release-target-vs-evidence) @ 8bcf674}

[gemini-code-assist]

Code Review

This pull request implements a mechanism to demote local replicas back to stubs under storage pressure when free space falls below the reclamation threshold, and adds reporting for replica counts and size estimates. Feedback on the implementation includes a critical issue in runReplicaSweep where returning a promise without await can prematurely reset the sweep state, an incorrect calculation in getReplicaStats where the walked counter is incremented outside the replica check (underestimating replica size), and a recommendation to use optional chaining when calling query?.get in the test resources to prevent potential TypeErrors.