Skip to content

A fork that a still-loading worker creates after drop_component's last census is never removed #3758

Closed

Description

@dawsontoth

What happens

dropApplication (components/operations.ts) decides whether the dropped app owes a fork removal by looking at what is on disk. It checks storage once before removeApplication and once after, and only then records the `removal-pending` marker that retryPendingBranchRemovals finishes from.

A fork that appears after the second check is never removed. A worker that has not yet reported what it loaded (poolMayHold) can still be inside prepareBranches for that app. If it creates the app's first fork after the drop's last check, no marker covers it:

  • The drop's answer says nothing about branches, because the app had none when it looked.
  • The retry ignores it, because there is no marker.
  • Nothing lists the app again, because it is absent. The preview expiry sweep does not either.

The fork stays on disk until someone drops the absent app by name. Nothing logs it.

Kris found this reviewing #3600; it predates that PR. It needs a load in flight during the drop, for example a worker booting while the drop runs.

Options

  1. Record the removal by declaration, not by what exists yet. When the dropped app's root-config entry declares branchedDatabases, write the marker into <storageRoot>/branches/<app>/ under every configured storage root, even with no fork there yet. A later fork lands beside the marker, and the retry removes it. The retry then has to keep a marker with no branches until no worker may hold the app, rather than clearing it at once. The cost is creating directories for apps that never forked.
  2. Refuse the fork in prepareBranches once the app is gone. Check that the component directory or root-config entry still exists before creating a fork, and fail the load otherwise. This narrows the window but does not close it: the check and the creation race, unless the load takes the component lock or the drop waits for loads in flight.
  3. Census again once every worker has reported (possible with Retry a branched database removal that drop_component could not finish, instead of only logging the failure #3600's follow-up). Main remembers apps dropped while a worker had not reported. On each onWorkerRelease, from the follow-up that wakes owed removals when a worker exits or reports, it checks those apps again and records a marker for any fork that appeared. It forgets them once no unreported worker remains. Only an unreported worker can still create a fork, so this closes the window without durable state. A crash restarts every worker on code that no longer has the app.

Part of #3042.

🤖 Generated by Anthropic Claude (Opus 5.5); posted via @dawsontoth.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Priority

    P3

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions