Recompile failure on source delete leaves page live with stale content

[tuirk]

Scenario

A wiki page is built from 3 sources. You delete source #1. The system needs to rewrite the page using only sources #2 and #3, so it no longer reflects source #1's material. That rewrite calls Gemini — an external API that can fail.

Outcomes today

Rewrite succeeds: page gets fresh content from sources #2 + #3. ✅

Rewrite fails (Gemini timeout, rate limit, transient network, etc.) — app/src/app/api/sources/[source_id]/route.ts#L281-L298:

• Leaves the old page body untouched
• Appends a footer: Source "X" deleted on YYYY-MM-DD. Rewrite failed.
• Decrements source_count by 1
• Logs page_recompile_failed

The page stays visible in the wiki — same title, same position, same content. The only hint anything went wrong is the footer line.

Why this is a problem

The page body still contains material from the deleted source. The user asked the system to remove that source, but its content is still on the page:

• Chat retrieval will still surface it
• FTS search will still return it
• Graph view still shows it live
• Related-pages still link to it

Proposed behavior — "archive on failure"

Instead of leaving the page live with stale content, hide it:

• Flip page_type to archived (or let the derived-archive mechanism handle it)
• Page disappears from the default wiki view
• Still reachable via "Show Archived" toggle
• Next successful recompile (manual trigger or retry job) restores it

Simpler framing: "can't rewrite it right now → hide it until we can" instead of "can't rewrite it → leave the wrong version up and hope someone notices the footer."

Open design questions

1. Archive on first failure, or retry first? Archiving immediately is safer but annoying on transient Gemini blips. Retry-with-grace-period needs a retry queue that doesn't exist yet.
2. What triggers the un-archive? Manual recompile only, or a background retry job?
3. Visible degraded badge as a middle ground? Keep page live but stamp a warning banner that the content is stale. Less aggressive than archiving, more visible than a footer.

Related

• The successful recompilePage path already archives correctly when no sources remain (app/src/lib/recompile.ts#L125-L129).
• Short-source delete (< 500 chars, >= 2 remaining) intentionally skips recompile — not in scope here.
• Single-source delete (remainingCount <= 1) hard-deletes the page — working as intended.

Status

Known issue. Deferred pending design decision on #1–#3 above.

[tuirk]

2026-04-30 — Verified still reproducible + revised plan

Verification (still open)

Catch block at app/src/app/api/sources/[source_id]/route.ts#L281-L298 is unchanged in shape since this issue was filed. None of the three originally proposed approaches have shipped. What has changed is adjacent: recompilePage now returns {outcome: 'rewritten' | 'archived'} and the success path acts on it (route.ts#L256-L280), which addresses the zero-remaining-sources edge case but not the failure path.

The actual user-visible bug (reframed)

The original report listed four surfaces where deleted-source material still leaks: chat retrieval, FTS, graph, related-pages. Chat retrieval is the loudest — it's the most likely place a user notices the bug, and "I deleted X but the assistant still cites it" is the literal complaint. Any fix that doesn't touch chat retrieval ships a half-fix.

Rejected alternatives

• Archive on first failure (original option 1): rejected — a single Gemini 503 blip wipes a healthy page from the wiki for a problem the user might be fine with. Strips user agency.
• Per-card STALE pill in /wiki (initial swarm draft): rejected — 30 amber pills after a Gemini outage = noise, not signal. One top-strip banner with count works better.
• recompile_queue table + drain hooks (initial swarm draft): rejected for v1 — yearly-event failure on a single-user app. Build it when incidence rate justifies it.
• Graph node dimming: rejected — perceptually invisible at typical node counts, no concrete spec.
• `page_recompile_queued` / `page_recompile_recovered` activity events: rejected — existing page_recompile_failed covers it; reconcile state on the page itself.
• Per-attempt activity logging: rejected — log per-degrade event only, per-attempt goes to console.error.

Revised plan

Mechanism (one sentence): retry-with-elapsed-budget around Gemini calls; on exhaustion mark pages.recompile_status='degraded', filter degraded pages out of chat retrieval, surface one banner on the page detail view with three real user actions: Rewrite now / Keep as is / Delete page.

Schema — migration v21:
```sql
ALTER TABLE pages ADD COLUMN recompile_status TEXT;
-- NULL = healthy. 'degraded' = body has stale content from a deleted source.
-- 'recompiling' = mutex held by an active recompile.
CREATE INDEX idx_pages_recompile_status ON pages(recompile_status)
WHERE recompile_status IS NOT NULL;

-- Backfill pre-existing failures so users see them post-migration:
UPDATE pages SET recompile_status='degraded' WHERE page_id IN (
SELECT json_extract(details,'$.page_id') FROM activity_log
WHERE action_type='page_recompile_failed'
);
```

Code changes:

1. app/src/lib/recompile.ts: add `withRetry({maxElapsedMs: 60_000, baseMs: 1000, factor: 2})`. Retryable: 429/500/503/504/timeout/ECONNRESET. Non-retryable: 503-with-detail `daily_cost_ceiling` (this is our internal cap mapped to 503 at recompile.ts#L74, retry burns budget against a wall). Refactor `recompilePage(pageId, opts?: {removedSourceId?: string})` so backfilled rows can manually retry without a removed_source_id.
2. app/src/lib/db.ts: add `markPageDegraded`, `markPageHealthy`, `tryAcquireRecompileLock` (single mutex for all entry points — UPDATE…WHERE status IS NULL OR 'degraded'), `getDegradedPageIds(): Set`. Patch `clearPendingContent` to also clear `recompile_status` atomically (Phase-3a outbox flush recovers the degrade flag).
3. app/src/app/api/sources/[source_id]/route.ts#L281-L298: catch → `markPageDegraded` + log + counter. Drop the `addProvenanceNote` "Rewrite failed" footer (was the bug's confusion vector). Short-source path keeps its footer (different semantics).
4. Chat retrieval boundary: add `excludePageIds = getDegradedPageIds()` to FTS5 + vector search, mirroring the existing `getAllArchivedPageIds()` exclusion at db.ts#L579. This is the change that closes the bug.
5. app/instrumentation.ts: boot reconciler unconditionally clears `recompile_status='recompiling'` (nothing in-flight at boot).
6. app/src/app/api/health/route.ts: add `markStaleRecompileLocks(5)` next to existing `markStaleSessionsFailed(30)` and `reconcileStuckCompileSessions(5)` — 5-min runtime TTL only, mirrors existing pattern.
7. Manual retry: extend existing /api/sources/[source_id]/recompile/route.ts for page-id rewrite, or add sibling. No new infra.

UI:

• Page detail banner: "Some content on this page came from a source you removed. We can rewrite from what's left." [Rewrite now] [Keep as is] [Delete page]. Reuse existing RetryButton.
• Top-strip on `/wiki` when any degraded: "N pages need rewriting → [Review]". No per-card pills.
• No graph treatment.

State machine:

State	wiki list	wiki detail	graph	FTS/chat	Trigger
healthy (NULL)	normal	normal	normal	normal	delete fails recompile → degraded
recompiling	normal	normal	normal	normal	resolves in same request
degraded	normal + counted	banner	normal	excluded	Rewrite/Keep/Delete or all-sources-gone (→ archived)
archived	hidden	hidden	hidden	excluded	(existing)

Test scenarios:

1. Gemini 503 once → succeeds on retry → no degrade.
2. Gemini 503 >60s → degraded, banner renders, chat retrieval excludes.
3. Manual Rewrite → success → flag cleared.
4. Manual Keep as is → flag cleared, body untouched, activity records the explicit accept.
5. Manual Delete → page gone, no orphan.
6. Phase-3a write fail → `pending_content` + degraded; boot reconciler atomically clears both via patched `clearPendingContent`.
7. `daily_cost_ceiling` 503 → no retry, immediate degrade.
8. Concurrent retries on same page → second bails (mutex rowCount=0).
9. Crash + Docker auto-restart → boot unconditional lock clear; next manual retry works.
10. Migration v21 backfill: pre-existing failed pages become degraded post-migration.

Explicitly accepted risks:

• No background drain — if user never re-opens a degraded page, it stays degraded. Acceptable: chat is filtered, so the bug doesn't surface anywhere user-facing. Revisit with a queue + drainer if incidence exceeds ~5/month with real data.
• "Keep as is" preserves a body containing deleted-source material. Deliberate user choice, recorded in activity log.

Effort

~3 days vs. ~7 for the original full-queue plan.