Batch-level consecutive-failure abort policy

[rcbevans]

I run batches of jobs that call a paid, occasionally-flaky external API. When that API goes down, the current failure shape is the worst one: every job in the fleet burns its full retry budget — max_attempts with backoff, per job, spread across workers — and then the batch fails anyway. So I pay for N × max_attempts doomed API calls and the batch takes the longest possible time to arrive at the outcome the first handful of failures already predicted.

Retry is strictly per-job, and the batch primitives (BatchHandle.status, wait_for_batch, BatchCompletionStatus) are read-only aggregation. There's no shared "if K jobs in this batch/queue have failed consecutively, stop dispensing and fail the rest" — no counter keyed by batch or queue that the dispatch path consults, no abort-the-remainder primitive.

Today I approximate it from a supervisor process: poll BatchHandle.status, and once failed crosses a threshold, paginate through the remainder and cancel one job at a time. Jobs keep dispatching between polls, and the abort decision lives outside the fleet, so every app that needs this re-implements it (slightly differently, slightly wrong).

What I'd rather declare, roughly:

await client.enqueue_batch(
    items,
    failure_policy=AbortBatchAfter(consecutive_failures=5),
)

or the same idea at the actor_config/queue level — a shared consecutive-failure counter (the leader loop or the dispatch transaction feels like the natural home) that, past the threshold, moves the batch's remaining pending jobs to a terminal state without dispatching them. Snooze/RetryAfter already give per-job control flow; this would be the batch-level analogue.

If this is achievable today through some hook I've overlooked — the reclaim event cursor, a leader-side extension point — I'd love a pointer. Otherwise: does a batch/queue failure policy belong in TaskQ, or do you see this as firmly application-side orchestration?

[rcbevans]

One clarification on the counter semantics, because there's a definition here that's easy to get wrong and changes the threshold's meaning entirely.

Count terminal job failures, not attempts. If the counter increments per failed attempt, then with max_attempts=3 a threshold of 5 trips after two jobs have exhausted their retries — the policy fires while the batch has barely started, and the effective threshold silently depends on the retry policy. Counting only jobs that reach a terminal failed state makes the threshold mean what it reads as, and keeps it independent of max_attempts.

"Consecutive" also needs pinning down across a fleet. With N workers draining a batch in parallel there's no meaningful serial order to be consecutive in — interleaving alone can produce success, fail, success, fail and never reach 5 in a row even while half the batch is failing. Two options that do work: a monotonic counter reset to zero on any terminal success (cheap, and what "consecutive" naturally means once you stop trying to order the failures), or a windowed ratio over the last K terminal outcomes. The first is simpler and probably enough for the flaky-dependency case that motivates this.

A ratio needs an absolute floor. If the policy is ever expressed as a share rather than a count — "abort if more than 20% failed" — then on a 3-item batch a single failure is 33% and aborts a batch that was fine. Anything ratio-shaped wants a minimum absolute failure count alongside it.

Lastly, aborting the remainder is the same set-based write as #54: pending/scheduled rows matching the batch move to terminal without dispatching. If bulk cancel lands first, this becomes a policy that calls it rather than new machinery — worth deciding which is the primitive and which is the caller.

[rcbevans]

One more, related to the "count terminal failures" point above: which terminal states count needs to be explicit, because there are more of them than the obvious one.

BatchCompletionStatus reports succeeded, failed, cancelled, crashed and abandoned. A failure policy — or any health check built on this shape — that reads only failed will score a batch whose workers were OOM-killed as completely healthy, since those jobs land in crashed, not failed. Same for lease abandonment. That's the case where an abort policy is most valuable and least likely to fire.

cancelled probably wants excluding, though, and for a reason specific to this feature: if the policy's own action is to cancel the remainder, then counting cancelled toward the failure total makes the policy self-feeding — it trips, cancels, and its own cancellations then satisfy the threshold again. Worth pinning down as part of the semantics rather than leaving to each caller.

So the natural default looks like failed + crashed + abandoned, with cancelled treated as an operator/policy action rather than a failure. If that's right, it'd be worth stating on BatchCompletionStatus itself, since anyone hand-rolling a health gate today on .failed alone has the same blind spot whether or not this policy lands.