ci: dump rancher server logs alongside webhook logs on CI failure - #1777
Open
tomleb wants to merge 5 commits into
Open
ci: dump rancher server logs alongside webhook logs on CI failure#1777tomleb wants to merge 5 commits into
tomleb wants to merge 5 commits into
Conversation
The current failure-log dump only captures rancher-webhook's own logs and its webhook configurations. When webhook never gets deployed at all (e.g. rancher itself fails to come up, or the systemcharts controller never installs the chart), that gives us nothing to work with. Expand the log dump to also cover: cluster-wide pod status, events in cattle-system, helm releases, apps.catalog.cattle.io status, the rancher pod's describe+logs (current and previous), the transient helm-operation install-job pods' logs, and rancher-webhook's describe + previous-container logs. Signed-off-by: Tom Lebreux <tom.lebreux@suse.com>
Adds a probe-assets-image-regression job (matrix of 2, not gating the PR) that stands up cert-manager + rancher alone (no webhook image imported) from two pinned rancher/rancher commits in parallel: - 3d7396a2b7 (immediate parent of PR #56253, the assets-image init container change) - 5f868f27e4 (PR #56253 itself, 'Implement Rancher assets image') This directly tests whether #56253 alone is sufficient to reproduce the startup-probe/git-index-lock failure that's been blocking webhook CI, or whether a later change (e.g. #56261, 'fix: checking out bundled charts') is also required to trigger it. Signed-off-by: Tom Lebreux <tom.lebreux@suse.com>
Repo was added as 'cert-manager' name pointing at jetstack's URL in the original CI script this was adapted from, but I renamed it to 'jetstack' when adapting it for the probe job while leaving the 'cert-manager/cert-manager' chart reference unchanged, so the install failed immediately with 'Error: repo cert-manager not found' before rancher was ever touched -- both probe legs failed identically for this reason, not because of the regression under test. Signed-off-by: Tom Lebreux <tom.lebreux@suse.com>
Round 1 (parent commit vs #56253 alone) showed #56253 is not sufficient by itself to fail the startupProbe in isolation, though it more than doubles rancher's startup time (2m52s -> 6m36s). Round 2 adds: - 1b96837a0a (#56261, 'fix: checking out bundled charts') in isolation, to see whether the added git reset --hard on bundled catalogs is what actually enables the restart-triggered index.lock corruption. - f74f4fdb55 (Aug 13, includes both #56253 and #56261 plus later changes) as a closer proxy for what current main's real CI job is running against. Signed-off-by: Tom Lebreux <tom.lebreux@suse.com>
5d1a1a2f12 (Aug 14, 'Remove annotation logic from GR and GRBs') is an ancestor of cf457b053f -- the exact commit the real webhook CI job's rancher/rancher checkout built from in the failing run -- and includes both #56253 and #56261. This gets the probe as close as practical to what the real CI job's rancher instance is actually running, short of pinning to the literal cf457b053f commit itself. Signed-off-by: Tom Lebreux <tom.lebreux@suse.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The CI failure-log dump step only captured
rancher-webhook's own logs and its admission webhook configurations. When webhook never even gets deployed (e.g. rancher itself fails to start, or the systemcharts controller never installs the chart), that leaves nothing to diagnose the actual root cause.This expands the failure-log dump to also capture:
kubectl get pods -A -o wide)cattle-system(sorted by time)cattle-systemapps.catalog.cattle.iostatus (shows systemcharts install/deploy state)rancherpod describe + logs (current and--previous, in case it crashed/restarted)helm-operation-*job pods (the systemcharts chart-install jobs)rancher-webhookpod describe +--previouslogsEach section is wrapped in a
::group::/::endgroup::for readability in the Actions log viewer.Test Plan
if: failure()log-dump step; it doesn't change any other CI behavior. Will be exercised for real the next time CI fails, which given the current failing state of CI it should be very soon.