Skip to content

A failed deploy payload whose node left the cluster is never expired #3718

Open

Description

@dawsontoth

Part of #641. Left over from #2681 / #3502.

Problem

sweepFailedPayloads (components/payloadRetention.ts) enforces failedMaxCount / failedMaxAge by ranking every node's failed rows per project, but each node drops only rows it wrote (row.origin_node !== hostname() skips the rest), so each row is written once. A failed row whose origin node has since left the cluster never matches any remaining node, so its payload blob is kept on every node forever. That's the unbounded growth #2681 was meant to end, now limited to clusters that have removed a node.

Proposal

Pick one surviving node to drop rows whose origin_node is no longer a cluster member, deterministically (for example the lowest-sorted current node name), so the row is still written once. A departed node's rows then follow the same bounds as everyone else's.

Done when

After the node that wrote it is removed from the cluster, a failed payload past the retention bounds is dropped, by exactly one remaining node, and every node unlinks its copy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Priority

    P3

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions