ci: run bare metal maintenance on the right nodes - #2597
Open
sespiros wants to merge 2 commits into
Open
Conversation
sespiros
added
no changelog
sespiros
force-pushed
the
sse/bm-maintenance-runner-host
branch
from
August 20, 2026 20:37
14cd4f1 to
33599a7
Compare
The job garbage collects the nix store of the machine hosting the GitHub runner, but selected ci.contrast.edgeless.systems/main-runner=true, which two nodes have carried since a second bare-metal machine joined the TDX-GPU cluster on 2026-08-14. That node needs main-runner so test pods can be scheduled on it for B300 GPUs (#2566), so pin nix gc to the runner's own machine instead, and fail with a named error when that label is missing rather than after a 600s wait. It has timed out three times since, on 2026-08-18 and twice on 2026-08-20. Signed-off-by: Spyros Seimenis <sse@edgeless.systems>
sespiros
force-pushed
the
sse/bm-maintenance-runner-host
branch
from
August 21, 2026 07:26
33599a7 to
7108310
Compare
cleanup and cleanup-containerd select main-runner=true, which two nodes have carried since a second bare-metal machine joined the TDX-GPU cluster. They still pass, but each run cleans one of the two and leaves the other's runtime class directories, snapshotter state and images behind. Run the same job once per schedulable matching node, and fail when none is labelled rather than quietly cleaning nothing. The job is deleted before each apply because spec.template is immutable, so applying the next node's variant over one left by an interrupted run would fail. The job timeout grows with it: the worst case is one 600s wait per node. Signed-off-by: Spyros Seimenis <sse@edgeless.systems>
sespiros
force-pushed
the
sse/bm-maintenance-runner-host
branch
from
August 21, 2026 08:46
7108310 to
9daf20c
Compare
sespiros
marked this pull request as ready for review
August 21, 2026 09:32
charludo
approved these changes
Aug 24, 2026
charludo
left a comment
charludo
left a comment
Collaborator
There was a problem hiding this comment.
Thanks! In general, LGTM. I'm not overly familiar with the available labeling strategies (both technically and best-practice) though, maybe Markus should also take a look at this?
| echo "::error::no node labeled ci.contrast.edgeless.systems/main-runner=true" | ||
| exit 1 | ||
| fi | ||
| while IFS= read -r node; do |
Collaborator
There was a problem hiding this comment.
(sorry, sanity check: we expect nodes to always have len==1, right? otherwise I'd comment that a failure on one node means we never try the other nodes)
| kubectl delete job -n maintenance-cleanup cleanup-maintenance --ignore-not-found | ||
| sed -e "s#@@REPLACE_IMAGE@@#${IMAGE}#g" -e "s#@@REPLACE_K3S_VERSION@@#${K3S_VERSION}#g" -e "s#@@REPLACE_NODE@@#${node}#g" \ | ||
| ./tools/bm-maintenance/cleanup.yml | kubectl apply -f - | ||
| kubectl wait -n maintenance-cleanup --for=condition=complete --timeout=600s job/cleanup-maintenance |
Collaborator
There was a problem hiding this comment.
I wonder if we can race a second kubectl wait with --for-condition=failed on all of these and have them cancel the attempt long before the 600s timeout in case of an ctual error
| Once the node has joined the cluster, apply the label with: | ||
|
|
||
| ```bash | ||
| kubectl label node <node> ci.contrast.edgeless.systems/runner-host=<runner name> |
Collaborator
There was a problem hiding this comment.
Can we just use kubernetes.io/hostname=<runner name> and not have to worry about labeling the hosts? Or is there a downside
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two nodes have carried
ci.contrast.edgeless.systems/main-runner=truesince a second bare-metal machine joined the TDX-GPU cluster on 2026-08-14, so the maintenance jobs pick one at random.nix-gccollects the runner's nix store and dies on the other machine (timed out 08-18 and 08-20, blocking v1.23.1, hence b0753f7).It now selects
runner-host=<runner name>, which names exactly one node.cleanupandcleanup-containerdpass but only clean the node they land on, so they now run once permain-runnernode.Before merging, label each runner's node in its own cluster, or nix-gc goes Pending and times out:
for
dgx-007,palutena,discovery,olimar,hetzner-ax162-snp.Test-target nodes like the second TDX-GPU machine keep
main-runnerand get norunner-host.Revert b0753f7 once the
bare metal maintenancerun this merge triggers is green on TDX-GPU.Why nix-gc dies on that node is still unknown: the job deletes its namespace on failure before anything captures the pod state.