Skip to content

Graceful shutdown doesn't await scope 'close' cleanup — app teardown races process.exit #1912

Open

Description

@heskew

Summary

The supported pattern for application shutdown cleanup is a 'close' listener on the plugin-API Scope:

export function handleApplication(scope) {
	const service = startService();
	scope.once('close', async () => {
		await service.close();
	});
}

But nothing in the shutdown path waits for that cleanup. Async 'close' listeners race process.exit, and on an idle server the cleanup may not even start before the process exits. This came up as a support question from an app developer; the recommended answer only works best-effort today.

Mechanics (all on the worker thread, triggered by the SHUTDOWN ITC message)

  1. scope.close() is fire-and-forget. componentLoader.ts:472 registers onMessageByType(ITC_EVENT_TYPES.SHUTDOWN, () => scope.close()), and notifyMessageListeners (server/threads/manageThreads.js:610-616) invokes typed listeners synchronously, dropping returned promises.
  2. Async 'close' listener promises are dropped. Scope.close() (components/Scope.ts:181-187) awaits entry-handler and options teardown, then calls plain this.emit('close') — EventEmitter.emit does not await listener promises.
  3. Exit is sequenced only on HTTP drain. threadServer.js:175-177 does closeServers().then(() => realExit(0)), where realExit is the original process.exit. closeServers() resolves when every server's close() callback fires (or a 5s per-server cap) — on an idle server that's near-immediate.
  4. Because scope.close() awaits entry-handler/options teardown before emitting 'close', on a fast drain the 'close' event may fire after realExit(0) has been queued — cleanup never starts.
  5. The backstop timers (threadTerminationTimeout = 10s worker-side at manageThreads.js:655-663, 2× on the main thread before worker.terminate() at manageThreads.js:296-306) are upper bounds only — nothing ever waits for scope cleanup, so finishing under the timeout doesn't help.

Note this affects graceful restarts too (dev watch, restart operation) — same SHUTDOWN message path via restartWorkers.

Impact

Apps that flush buffers, drain queues, or close connections to external services in a 'close' listener can be cut off mid-flight (or never run) on shutdown/restart. Typical symptoms: lost telemetry/batches, unclean client disconnects, external resources left open.

Suggested direction

  • Make the worker's exit condition Promise.allSettled([closeServers(), allScopeCloses]) (bounded by threadTerminationTimeout) instead of closeServers() alone.
  • Have Scope.close() await async 'close' listener results — emit can't do this, so either iterate rawListeners('close') and await returned promises, or add an explicit registration API (e.g. scope.onClose(async () => ...)) and document 'close' listeners as sync-only.

Docs follow-up (documenting the cleanup pattern + this caveat) tracked separately in the documentation repo.

Remaining loading/disposal liveness variant

The TypeStrip integration now awaits scope cleanup and separates load-verdict reporting from admission. A retained variant remains when SHUTDOWN unrefs the parent port while pending component loading or disposal completes only through an unreferenced callback: the worker's event loop can end before teardown finishes. This pass code-traced the variant at server/threads/manageThreads.ts:3226 and server/threads/threadServer.ts:278-292 on f3f3d9af630c8e41d57415da8c5b49f0374d4575; it did not reproduce it afresh. The current held-scope fixture uses referenced disposal timers, so it does not establish this case. A discriminating follow-up should require disposal to precede exit with an unreferenced completion source in both runtimes. Admission itself no longer blocks teardown.

Review-Deferral: c8fea45aee59d5a7 severity=minor category=reliability

🤖 Generated by OpenAI Codex; posted via @kriszyp.

Main-thread component loading during process exit

Review of Add opt-in native full-text analytics #3946 identified a separate, code-traced residual in the graceful shutdown code merged by Drain in-flight work and close cleanly on SIGTERM/SIGINT/SIGQUIT, bounded by threads.shutdownTimeout #3943. In multi-worker mode (threads.count > 0), main awaits loadRootComponents() in server/threads/socketRouter.ts, while shutdownForExit() in server/threads/threadServer.ts waits a componentsLoading promise assigned by the worker/single-thread startup route. If main receives a signal while component loading is unresolved, shutdown can sweep deferred scope closes before loading creates its last scope. A close registered after that sweep has no subsequent sweep before process exit.

This is a code-traced possibility, not a reproduced failure or an analytics regression. Add a deterministic main/multi-worker fixture that pauses component loading, initiates signal shutdown, creates a final scope, and requires that scope's async disposal to settle before exit under the existing deadline. Confirm the loading promise and disposal boundary share ownership before choosing a fix; preserve the already-correct worker and single-thread routes. No production incident or data loss is established.

Review-Deferral: e00b38df16d81c48 severity=nit category=reliability

Comment generated by kAIle (GPT-6)

Activity

  1. added this to the v5.2 milestone on Aug 6, 2026
  2. added theissue type on Aug 6, 2026
  3. kriszyp commented on Sep 27, 2026

    @kriszyp
    Member

    The premise here does not hold for the primary shutdown path, which makes the bug broader than "the await races process.exit". Verified on origin/main (2026-09-03).

    bin/run.ts's addExitListeners() wires SIGINT/SIGQUIT/SIGTERM to:

    const exit = () => { beginProcessShutdown(); removeHdbPid(); process.exit(0); };

    beginProcessShutdown() (server/threads/manageThreads.js:797-799) is a one-line flag setter — processShuttingDown = true. It does not call shutdownWorkers / restartWorkers, which is the only code that ever does worker.postMessage({ type: SHUTDOWN }) (manageThreads.js:637). The sole caller of shutdownWorkersNow() in the whole repo is bin/restart.ts:89, i.e. the restart-service path.

    So on a real OS SIGTERM / docker stop / killHarper, the main thread calls process.exit(0) directly — terminating every worker thread immediately — and SHUTDOWN is never posted to any worker. Scope 'close' listeners do not race the exit; they never run at all, sync or async, at any delay. This issue's analysis (SHUTDOWN is delivered, then the async scope.close() loses the race) describes the restart/dev-reload path only.

    One correction to this issue's own citation while we're here: componentLoader.ts:472's fire-and-forget scope.close() is stale — d72d5b035 (2026-06-16, pre-dating this issue) plus components/componentLoader.ts:1002 already wrap it in trackScopeClose(...). The awaiting half is done; the signalling half on the real shutdown path is not.

    Practically this means the documented app-teardown hook does not run in the way apps actually stop in production: a container stop.

    (qa-explorer campaign, F-185.)

    — Claude Opus 5.5

  4. modified the milestones: v5.2, v5.4 on Oct 22, 2026
  5. kriszyp commented on Oct 23, 2026

    @kriszyp
    Member

    A retained startup-specific variant from #562 review belongs with this shutdown-liveness issue. At 386f55f972e482ee56736c1215c8251f51535dd8, server/threads/manageThreads.ts:3226 unrefs parentPort when SHUTDOWN arrives, while server/threads/threadServer.ts:278 waits for the loading promise before running disposal. A loading step with no referenced handle can let the worker exit before cleanup. A preparation-lock wait can also outlast the worker's exit backstop.

    The branch's both-mode unreferenced-timer probe passes; it does not establish liveness for every native-storage wake or a load held beyond the deadline. Main controls also fail shutdown during loading, so this is retained as an explicitly limited lifecycle contract during the TypeStrip merge, rather than a claim that the merge repairs every shutdown race. The earlier review groups are 917206da96da, 250770341ee4, 1126b433b3f1, c86e355ec346 and a2be04d0875d.

    Please include a cold-worker regression that gates real component preparation, sends SHUTDOWN, and verifies disposal precedes exit both when the pending step has no referenced handle and when preparation exceeds the backstop. Keep this separate from late listener creation after close (#3472) and the process-level signal path (#3452). No shipped-version boundary or production incident is established by this source-derived report.

    🤖 Generated by OpenAI Codex; posted via @kriszyp.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Priority

    P2

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions