Repository navigation
fix: Recover NATS receiver after broker outages - #266
Merged
Merged
Conversation
A broker outage longer than nats.go's default reconnect window (60 tries, 2s apart) closed the connection for good. The receiver set no closed handler and reported nothing, so it stopped consuming while the collector stayed healthy. A consumer deleted on the server, or missing when the broker came back, ended the same way. Key features/changes: - Reconnect without a limit, so a broker replacement of any length recovers by itself - Report a fatal status to the collector, so it exits and Kubernetes restarts it, when the connection closes after a successful Start, when JetStream ends the consume with consumer deleted or bad request, or when a missed heartbeat finds the consumer or stream gone - A lookup that fails for another reason, such as the broker still being down, stays a warning - Install the closed handler at connect, and report nothing before Start succeeds or once Shutdown begins - Close instead of drain when Shutdown runs mid-reconnect - Add tests for unlimited reconnects, a closed connection, a deleted consumer, a consumer missing after the broker returns, an outage that must not report fatal, and a clean shutdown
ecv
changed the title
ecv
force-pushed
the
fix/natsreceiver-self-recovery
branch
from
November 9, 2026 13:07
93c1901 to
ac02cf8
Compare
ecv
marked this pull request as ready for review
November 9, 2026 13:20
ecv
enabled auto-merge
November 9, 2026 13:20
Contributor
Author
|
Review loop: adversarial and conventions reviews returned merge. Agreed fixes applied at ac02cf8 (bad request and missing consumer now fatal, closed handler set at connect) and confirmed by re-review, including tamper tests. The conventions re-review could not run Go locally; CI covers it, 30 of 30 checks green. Residual risk: a newly elected JetStream leader might briefly report a consumer missing during a rolling restart, which would cost one extra collector restart. The staging check after release covers it. |
andrewb3000
approved these changes
Nov 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
After a NATS broker outage of more than about two minutes, the NATS receiver stopped consuming for good while the collector kept reporting healthy, so logs piled up until someone restarted the pod by hand. Now the receiver keeps reconnecting until the brokers return and resumes by itself. If the connection still closes, or the stream consumer is deleted or missing when the brokers come back, the collector exits so Kubernetes restarts it.
Test plan
Fixes #265