[kriszyp]

⊙ Problem

Every new PR-preview deploy (isolated: true, restart: true) restarts every shared worker on every node, and its first release is not canary-certified. Cleanup is worse: drop_component leaves the preview's worker running unless restart: true, restarts the shared workers when the app is not on a node, and reports nothing per node.

🤔 Should #3045 and #3091 ship as one PR?

Yes: both change the same drop path, and one drop to the cluster URL helps only once it both stops the worker and reports per node. Docs: Document starting and stopping one isolated application, and per-node drop results (documentation#750).

💡 Solution

• A new isolated app's first restart: true deploy starts only its own worker. That worker is the canary: a release that fails to load leaves no worker and no socket.
• Dropping an isolated app stops only its worker, with or without restart, and removes its branched databases unless keepBranches: true.
• Dropping a component a node does not have restarts nothing there.
• drop_component answers { node, componentRemoved, workerStopped, branchesRemoved }; replicated carries the same per peer, plus error.
• Turning isolation on or off still restarts the shared workers.

⚖️ Alternatives

• Guess which shared workers hold an app from config and the directory — a worker can load an app that appeared after it started; each worker now reports what it loaded.
• Stop an isolated app with a scoped restartWorkers — that reloads every root component on main; main never loads isolated apps.

Also weighed

• Fail the drop when a peer fails — drop never did; the per-node error shows it.
• Report a fork-removal failure to a worker that dropped itself — it is stopped first; the failure is logged, and a second drop retries. Tracked in A fork-removal failure after drop_component answers early is only logged (#3463).
• Make the config refresh after a drop non-fatal — it exits on a config that parses but fails validation. Every main-thread deploy and drop already runs it, so that is a separate fix.

🤔 Should dropping an absent app still discard branches an earlier drop kept?

Yes, chosen: otherwise keepBranches, or a shared drop without restart, leaves storage no later drop can remove. Nothing restarts, and while a shared worker may still hold the fork it stays.

🤔 Should keepBranches be refused together with restart: true?

Chosen: an older peer ignores keepBranches, and its restart path removes the forks; without a restart it keeps them. A shared app with forks then drops and restarts in two calls; easy to relax later.

🤔 Is the node's own result at top level, with peers in replicated, the right shape?

Chosen to match deploy_component, where a peer returns the same object. The alternative: one nodes array including the origin.

🔧 Changes

• components/operations.js — deploy names an isolated app as the restart scope; the certification accepts a new isolated app. A whole-component drop runs on main and is answered by main for a worker; the worker hands it over and waits within its budget, and main refuses to start it late. Main reads isolation from the file rather than a refresh that exits on a bad config, removes the app, decides retention before any early answer with a reason in the message, and keeps the error class. File drops keep their path, plus the absent no-op.
• server/threads/manageThreads.js — ⚠️ Look hardest: an isolated release with no worker arms, and a refused first release skips its copy by leavesNothingToLoad, queues its slot's withdrawal before the decision publishes, and stays registered until it drains. Pool ownership comes from each worker's report via poolMayHold, read when the restart takes its turn, adding the pool to a named restart, which reconciles only that app. The request and answer helpers match replies by id.
• server/threads/socketRouter.ts — admission reports only for the reconciled app; failed-closed apps get no worker, and a forced withdrawal ignores a failed read; heap share counts real slots; the slot's ready bound covers a canary, and a withdrawn slot is not a failure.
• components/componentLoader.ts, server/threads/heldStart.ts — each worker records what it placed, deferred loads included, and reports it before admission.
• components/canaryRollout.ts — eligibility receives wasAbsent.
• components/operationsValidation.js — keepBranches: a strict boolean; keepBranches: true is refused with restart: true.
• bin/restart.ts — the worker-drop branch-removal path main now owns is removed.
• utility/hdbTerms.ts — two ITC message types.
• Design notes — the restart-scope invariant, the main-owned drop, arming with no worker, no worker for a failed-closed app, the load report, and their index entries.

✅ Verification

• New isolated-application-lifecycle.test.ts: 14 pass. Thread ids and mirrors across deploy, refusal (also across a restart), drops from main, a pool worker and the app itself, absent drops, both flips, the shared-drop-then-isolated sequence, and an unrelated pending app.
• branched-database-package.test.ts (+3 per placement, and a certified first isolated deploy): fork removed by default, kept with keepBranches, then discarded by a second drop. 12 pass.
• isolated-application.test.ts (7), canary-rollout.test.ts (10) and 9 new unit tests pass: the gate and main requests (with a fixture hook), eligibility, and peer results and validation. Every new test that checks new behavior fails on base.
• test:unit:main (7313), test:unit:resources (4407) and test:integration:all (2561; 6 Ollama cases cancel locally) pass.
• PR QA: skipped @ 59b7797 — dispatch unavailable (no worker token in the agent's environment); scenario drafted in the task handoff.

🤖 Generated with Claude Code

https://claude.ai/code/session_015gDmwTaExpc8uXcdBNr8eA

_{🤖 Generated by Anthropic Claude (Opus 5.5, dispatch dev-agent); posted via @kriszyp.}

Release target: main — new behavior and a new drop_component field, no linked milestone

Related PRs: 3 independent
Complexity: complicated

Deferred findings

• f7a59ff0061ac9c2 nit reliability — components/operations.js — branch-removal failure after an early answer only logged — NOT FILED

Origin — the dispatch brief this PR was written from

Implement #3045 (start and stop one isolated application without restarting the shared HTTP workers) together with #3091 (per-node results from a replicated drop_component; dropping an absent component is a no-op). One PR closing both. Part of epic #3042 (PR preview deployments): today every new PR preview's first deploy rolling-restarts every shared worker on every node, and cleanup has to call each node directly and verify it by hand.

Decided behavior:

1. DEPLOY of a NEW isolated application (component did not exist / was never loaded by any pool worker) with restart: true restarts ONLY its own dedicated worker: the restart scope is the application name, not undefined. It goes through the canary held start added by Certify a restarting deploy in a canary worker before it rolls out #2981 (merged), so the first deploy is canary-certified and the worker publishes its socket mirror only after a clean load. A flip of an EXISTING shared application to isolated (or back) still restarts the pool, because pool workers loaded it. Invariant to enforce: a restart's scope covers exactly the workers that loaded or will load the component.
2. DROP of an isolated application always stops only its dedicated worker and removes its socket mirrors, whether or not restart: true is passed. No pool restart.
3. DROP removes the application's branched databases by default (after its worker is stopped, as today's restart:true path does), with an opt-out request field (suggested name keepBranches: true) that retains them and keeps today's BRANCH_STORAGE_RETAINED message. Validate the new field in the drop validator.
4. A replicated drop_component returns a per-node result, { node, componentRemoved, workerStopped, branchesRemoved, error }, the way deploy_component reports its peers (onPeerResult / response.replicated). Dropping a component that is not present on a node (no component directory and no root config entry) is a successful no-op there and NEVER restarts anything — today dropComponent with restart: true falls to restartScope undefined and restarts the whole pool for an absent or shared-not-running component.
5. Replicated deploys and drops behave the same on every node (each peer computes its own scope from its own local state).

Acceptance

• Integration tests (multi-worker, and replicated where the harness supports it) proving: deploying a new isolated app with restart:true, and dropping it (with and without restart:true), leaves every other worker running with the same thread ids and no restart events, on the origin and on peers.
• The first deploy of a new isolated app is canary-certified; a failing load is refused and leaves no socket mirror and no running worker.
• Dropping an absent component (restart true and false) succeeds, reports componentRemoved:false, and restarts nothing.
• Dropping an isolated app with branched databases removes its branches by default; with the opt-out they are retained and the message says so.
• Replicated drop_component response includes one { node, componentRemoved, workerStopped, branchesRemoved, error } entry per node.
• Flipping an existing shared app to isolated still restarts the pool (regression test).
• Show each new test failing on origin/main sources (forced rebuild; integration tests run against dist, so npm run build first).
• Update operation docs/types for the drop response and new field. Do not change payload-deploy field handling (Payload deploys can set host, urlPath, isolated and branchedDatabases #3043) or deploy permissions (Scope deploy permissions to component names #3046).

_{Dispatch: task harper-3045 · queued by kris-session · ran by claude/opus/xhigh · worker kzyp-xps-1}

_{Review-Coverage: authored=claude; ran=gemini,codex; adjudicated=domain; declined=cursor-grok,cursor-composer,cursor-kimi,cursor-muse,muse; rounds=8; full=2 @ d37b2a1}

_{Review-Attention: deep ~25m (hot: heldStart.ts, manageThreads.js +1; decisions: do-less-alternative, test-retries-until-reported, config-order-reservation, error-identity-by-name, self-drop-waits-for-peers, keepbranches-refused-with-restart, withdrawal-fences-redeploy, default-branch-deletion, absent-drop-succeeds) @ d37b2a1}

[gemini-code-assist]

Code Review

This pull request refactors the component drop and restart lifecycle, ensuring that starting, stopping, or dropping an isolated application does not affect unrelated workers. It centralizes whole-component drops on the main thread under a component lock, introduces a keepBranches option, and implements targeted worker reconciliation. Feedback on these changes suggests enhancing robustness by safely handling nullish or primitive errors in catch blocks and IPC message handling, as well as wrapping secondary package.json updates in a try-catch block to prevent minor failures from disrupting the core drop operation.

[dawsontoth]

Yusss! I can run local tests of this tomorrow if you like, or merge and abuse it later, whatever works. Concept looks great!

[dawsontoth]

A new isolated app deployed from a worker thread never gets its worker, so every replicated peer misses it

Found while testing locally at d37b2a1 (macOS, Node 24.18).

What happens. A first deploy_component of a new isolated app with restart: true, run on an HTTP worker thread, answers 200 with Successfully deployed: <app>, restarting Harper and certification: "uncertified", but no dedicated worker starts. The app only comes up after a later restart_service. A replicated peer receives the operation on a worker, so this is the path every peer takes.

• Two harper-pro nodes, with core at this PR merged onto pro's pin 489eebcf0: a replicated first deploy of a new isolated app started its worker on the origin (certified). On the peer it did not, 4 times out of 4. The peer's entry in replicated said "certification":"uncertified", "message":"Successfully deployed: rep_app, restarting Harper".
• One core node, no replication: threads.count: 2, tls.unixDomainSockets: true, and a component whose resource runs server.operation({ operation: 'deploy_component', project, package: 'file:<tgz>', isolated: true, restart: true, replicated: false }). This reproduces every time. system_information shows only the two pool threads afterwards. On base 1fd591ba2 the same deploy does start the app's worker: base's scope there is undefined, so it restarts the pool, and that reconcile is what starts the worker.
• Not affected: a deploy through the operations API, which runs on main, and a worker-run redeploy of an isolated app main already knows.

Root cause (from temporary log lines in dist). At components/operations.js:923 the scope is now the app's name. Main commits the certification and reconciles only that app (server/threads/socketRouter.ts:296). presentIsolatedApplicationNames() reads main's in-memory config, which doesn't have the entry the worker just published yet:

PROBE arm       wd_one bb5125ab-… true wd_one
PROBE commit    wd_one bb5125ab-… armed wd_one true
PROBE reconcile {"only":"wd_one","refused":false,"wanted":[],"present":[],"slots":[]}
… 20 s later, a restart_service:
PROBE reconcile {"refused":false,"wanted":["wd_one"],"present":["wd_one"],"slots":[]}

With nothing wanted, no canary starts, and completeCertification decides uncertified ("no replaced worker loaded it").

Possible fix. The drop path already handles this case: configuredIsolated (components/operations.js:1808) reads isolation from the root config file as well as from this thread's view. The scoped reconcile, or commitCertification for a release a worker asked for, could do the same. Main could also refresh its config before committing.

Test gap. Every deploy in isolated-application-lifecycle.test.ts goes through the operations API, which runs on main in the harness. A first deploy through a worker resource, like the existing WorkerDrop, would have caught this.

The rest checked out locally:

• The PR's own suites pass.
• Replicated drops return one correct per-peer { node, componentRemoved, workerStopped, branchesRemoved, error } entry.
• Dropping an absent app restarts nothing on either node.
• A down peer gets an error entry, and the drop still answered in 56 ms.

🤖 Generated with Claude Code