Skip to content

Ctrl+C leaks function containers: no signal handling, so cleanup never runs on interrupt #515

Closed

Description

@jcogilvie

What happened?

Interrupting a run with Ctrl+C leaks the function containers it started. Nothing in cmd/ handles signals: there is no signal.Notify or signal.NotifyContext anywhere. So SIGINT (and SIGTERM, e.g. a CI job being cancelled) terminates the process with Go's default behaviour, and no deferred function runs, including the processor Cleanup that removes the containers.

The code believes otherwise. The cleanup defers in cmd/diff/xr.go and cmd/diff/comp.go both justify using a detached context this way:

The command context may be cancelled (user Ctrl+C, timeout, etc.), which would cause Docker API calls to fail immediately, leaving containers running.

Ctrl+C never cancels that context. The run context comes from initializeAppContext (cmd/diff/cmd_utils.go), which is context.WithTimeout(context.Background(), timeout), so the only way it gets cancelled is --timeout expiring. The Ctrl+C case the comments describe is the one case that always leaks.

CachedFunctionProvider names its containers with a random per-process instanceID, so a later run never reuses or reclaims them. Each interrupted run leaves its containers behind until someone removes them by hand. That matters most in CI, where cancelled jobs are routine.

How can we reproduce it?

crossplane-diff comp updated-composition.yaml   # any run that starts function containers
# press Ctrl+C while it is rendering
docker ps --filter name=crossplane-diff          # the run's containers are still running

How could crossplane-diff help solve your problem?

Derive the run context from signal.NotifyContext(ctx, os.Interrupt, syscall.SIGTERM) in initializeAppContext, so an interrupt cancels the run instead of killing the process. The existing cleanup path then runs on interrupt. It already detaches from the run context's cancellation, and it will use context.WithoutCancel once the shared cleanup helper in #496 lands, which is exactly what an interrupted run needs: the work stops, and cleanup still finishes within its own timeout.

Things to decide and test:

  • A second interrupt should force an exit. Otherwise a hung Docker daemon turns Ctrl+C into a 30-second wait (CleanupTimeout) with no way out. A common pattern: stop the signal handler after the first signal, so the next one uses Go's default and kills the process.
  • Exit code: an interrupted run should exit non-zero and distinguishably, not report success or "diffs detected".
  • Structured output: decide whether an interrupted -o json run still writes a document. The "always render" contract suggests it should, with an error saying the run was interrupted.

What environment did it happen in?

crossplane-diff main at 58d1ade. The behaviour is pre-existing. Found while reviewing the cleanup changes in #496.

Activity

  1. jcogilvie commented on Oct 4, 2026

    @jcogilvie
    CollaboratorAuthor

    Cross-linking the related upstream work in crossplane/cli. While validating this issue, we found and filed the same class of cleanup bugs in the render library that crossplane-diff builds on, and opened fixes for them.

    Upstream issues: crossplane/cli#396, crossplane/cli#397, crossplane/cli#398, crossplane/cli#399, crossplane/cli#400, crossplane/cli#401

    Upstream PRs (a stack; all drafts):

    PR Fixes What it changes Effect on crossplane-diff
    crossplane/cli#408 — Injects narrow Docker client interfaces into render (prep refactor) None directly; makes the fixes below testable
    crossplane/cli#404 crossplane/cli#397 Stops every runtime, with a graceful stop then a forced remove; StopFunctionRuntimes(ctx, fa) error Breaking signature change. EngineRenderFn's stopRuntimes seam needs a one-line update on bump. With this, the Orphan + named-container workaround in CachedFunctionProvider can be dropped
    crossplane/cli#403 crossplane/cli#396 Stops already-started runtimes when a later one fails to start Removes a leak EngineRenderFn currently inherits on a start failure
    crossplane/cli#405 crossplane/cli#398 Network cleanup logs its error instead of discarding it, using a detached, bounded ctx Network-removal failures become visible
    crossplane/cli#402 crossplane/cli#399 render.WarnCleanupFailure for cleanup failures Optional helper for the CLI layer
    crossplane/cli#406 crossplane/cli#400 signal.NotifyContext in crossplane's main; a second signal restores the terminal and exits 130; the Docker attach read now unblocks on ctx cancel The main change does not reach crossplane-diff, so this issue still needs its own NotifyContext. But the internal/docker attach fix does: without it, a cancelled ctx can't interrupt an in-flight render, which waits until the render container exits (~115s observed with a hung function vs ~0.3s with the fix). Its second-signal handling is a worked example for the "second interrupt should force an exit" point above
    crossplane/cli#407 crossplane/cli#401 Labels render's containers and networks (render.crossplane.io/managed-by=crossplane) Enables a label-based sweep for leftovers that no in-process handler can catch (kill -9, OOM, CI timeouts)

    So, for this issue: the in-process fix here is still needed. Once crossplane-diff bumps github.com/crossplane/cli/v2 past these, the interrupt path will actually unwind promptly, and the container-tracking workaround can go.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions