Repository navigation
fix: Restart wedged Vector and sink collector pods - #269
Merged
Merged
Conversation
The Vector exporter ran with its API off and no probes, and the sink collector had no health_check extension, so the OpenTelemetry operator generated no probes for it. A wedged process in either pod kept running and looked healthy. Vector 0.61 serves GET /health on its API port (8686) and the sink's health_check extension serves / on 13133; neither tracks a downstream sink, so an outage there fails no probe. Key features/changes: - Enable the Vector API on 0.0.0.0:8686 and probe /health for readiness and liveness - Add a NetworkPolicy admitting only the 9598 metrics port to Vector, since the same API port serves the unauthenticated gRPC event tap - Enable health_check on 0.0.0.0:13133 for the sink collector, with a two-minute startup window because the clickhouse exporter blocks startup for about 30s while ClickHouse is unreachable - Add config tests that fail on the pre-fix manifests
The Vector NetworkPolicy selector is correct only after the kustomization rewrites its labels. Nothing asserted that the rendered selector matches the Deployment's pod labels, so a label change would silently reopen the unauthenticated gRPC tap on port 8686 with every test green. Render the overlay and compare the two. Also pin the sink collector liveness settings (10s period, six failures). Vector /health source: https://github.com/vectordotdev/vector/blob/v0.61.0/src/api/grpc_server.rs#L158-L183 Vector API defaults: https://github.com/vectordotdev/vector/blob/v0.61.0/src/config/api.rs Operator probe generation: https://github.com/open-telemetry/opentelemetry-operator/blob/v0.162.0/internal/otelconfig/config.go#L322-L395 https://github.com/open-telemetry/opentelemetry-operator/blob/v0.162.0/internal/components/extensions/healthcheckv1.go
ecv
changed the title
ecv
marked this pull request as ready for review
November 10, 2026 17:02
andrewb3000
approved these changes
Nov 10, 2026
kevwilliams
approved these changes
Nov 10, 2026
kevwilliams
left a comment
kevwilliams
left a comment
There was a problem hiding this comment.
Verified Vector 0.61's /health tracks only shutdown, operator v0.144 honors startupProbe, and health_check's pipeline check and component-status gate stay off.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Vector telemetry exporter and the hub sink collector run with no health probes, so a wedged process in either pod keeps running and looks healthy. With this change both pods get readiness and liveness probes that check only the process itself, so a wedged pod is restarted while a NATS, ClickHouse or customer remote-write outage fails neither probe.
Fixes #268
Related to https://github.com/datum-cloud/infra/issues/9112
What changed
sidecarcontainer gets no probe from this change. A sidecar liveness probe restarts pods only if its threshold is shorter than a plausible API-server loss, so adding one here would restart every exporter during a control-plane blip. That is left to deployers; infra staging already has a lenient one.Test plan
Warning
Not verified on a cluster. The NetworkPolicy assumes the CNI admits kubelet probes from the node; GKE Dataplane V2 does this by default, but no probe has run against a deployed pod. The sink startup allowance of about two minutes equals infra's Flux two-minute timeout, so a ClickHouse outage during a rollout can fail the reconcile.