🐛 fix(civo/project-cluster): external-dns domain + one-shot Job hooks#53
Open
CristhianF7 wants to merge 77 commits into
Open
🐛 fix(civo/project-cluster): external-dns domain + one-shot Job hooks#53CristhianF7 wants to merge 77 commits into
CristhianF7 wants to merge 77 commits into
Conversation
* feat: add GPU cluster template and catalog items for Friday demo - Create templates/civo/gpu-cluster/ with isGpu=true and is_gpu Terraform var - Backfill is_gpu=false on workload-cluster template for consistency - Add nvidia-nim-operator catalog item (k8s-nim-operator v3.0.2) - Add runai catalog item (control-plane + cluster-agent v2.20.0) Closes #35, Closes #36, Closes #37, Closes #38 * feat: add konstruct.yaml metadata to gpu-cluster and workload-cluster templates Required for template registration via Konstruct UI/API. * fix: correct terraform module path in template default URLs The module lives under civo-github/, not at the repo root. * fix: update open-webui default chart version to 13.3.1 * fix: cert-manager template fixes from gpu cluster dry run - Downgrade cert-manager from v1.20.1 to v1.16.5 (selectableFields CRD incompatibility with older K8s versions) - Disable startupapicheck (hangs on Talos clusters) - Add ServerSideApply=true sync option (required for CRD apply) - Remove duplicate 35-cert-manager.yaml (caused ArgoCD warnings) - Add ServerSideApply to ollama-openwebui catalog app * fix: add ServerSideApply to NIM operator and Run.ai catalog apps Learned from dry run that CRD-heavy charts need ServerSideApply to avoid annotation size limits during ArgoCD sync. * fix: update Run.ai catalog to use public runai-cluster chart - Switch from JFrog control-plane chart to GCS runai-cluster chart (publicly accessible, no auth required) - Simplify to single Application (cluster agent only, no separate control-plane chart needed) - Update chart version to 2.16.79 - Replace domain/TLS config with cluster URL/UID inputs * fix(runai): restructure catalog for ArgoCD hook compatibility ArgoCD interprets helm.sh/hook annotations as hooks and blocks sync. Restructured Run.ai catalog to use pre-rendered manifests (hooks stripped) instead of direct Helm chart source. Catalog now outputs 3 Applications: - Parent app (wave 47): points at runai environment dir - Prereqs app (wave 49): creates runai, monitoring, runai-scale-adjust, runai-reservation namespaces - Manifests app (wave 50): deploys rendered runai-cluster 2.16.79 chart Static files in static/ dir are copied to gitops repo alongside rendered templates. Manifests use <RUNAI_CLUSTER_URL> and <RUNAI_CLUSTER_UID> token placeholders for cluster-specific values. * fix: set civo-gpu demo defaults in NIM and Run.ai catalog values * fix(nim): disable upgradeCRD hook in catalog template ArgoCD blocks on helm.sh/hook pre-upgrade Job. Setting operator.upgradeCRD=false prevents the hook from being rendered.
Add catalog templates for: envoy-gateway, external-dns, cert-manager, external-secrets, vault, prometheus (kube-prometheus-stack), grafana, reloader, and groundcover. Each template includes Chart.yaml, values.yaml with @input annotations for UI form generation, and an ArgoCD Application template. Closes #41
PSP was removed in Kubernetes 1.25+. The groundcover chart 0.36.3 creates PSP resources by default, causing sync failures on modern clusters. Disable pspEnabled across all subcharts (agent, loki, victoria-metrics, promscale, timescaledb) with a configurable toggle defaulting to false.
… Job hooks Three fixes for the civo/project-cluster template that left freshly-created mgmt clusters with Degraded/flapping ArgoCD apps: 1. external-dns domainFilters used .Values.domainName (default konstruct.io), never overridden, while all hostnames use .Values.domain (the real env domain that IS injected). external-dns published nothing (NXDOMAIN) -> ACME HTTP-01 failed -> certs unissued -> Gateway HTTPS listeners InvalidCertificateRef -> argocd/envoy-gateway-components/registry Degraded. Point domainFilters at .Values.domain and drop the now-unused domainName value. 2. github-token-refresher-init Jobs (argocd + crossplane-components) used hook-delete-policy: hook-succeeded, so ArgoCD deleted them on success while still tracking them as desired -> apps stuck OutOfSync/Missing. Use before-hook-creation so the completed Job persists. 3. apply-argocd-clusters-app was a plain Job with ttlSecondsAfterFinished: 100; k8s auto-deleted it ~100s after completion while ArgoCD tracked it -> the argocd app flapped Synced<->OutOfSync every ~100s. Make it a post-install/ post-upgrade hook (before-hook-creation) and drop the TTL.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three template bugs in
templates/civo/project-clusterleave freshly-created mgmt clusters with Degraded / flapping ArgoCD apps. Diagnosed live one2e-mgmt-cluster-k3m; hotfixed in the renderedk1-civo/gitopsand now fixed at the source here.Fixes
1. external-dns publishes nothing → NXDOMAIN → cert/Gateway cascade
templates/30-external-dns.yamlsetdomainFilters: [ {{ .Values.domainName }} ].domainNamedefaults tokonstruct.ioand is never overridden, while every hostname (gateway/HTTPRoutes) uses{{ .Values.domain }}— the real env domain that is injected at render (e.g.kubefunk.net). So external-dns filtered out all hostnames and published nothing:→ ACME HTTP-01 self-check fails → Certificates never issue →
Gateway/egHTTPS listenersInvalidCertificateRef→argocd,envoy-gateway-components,registryDegraded.Fix: point
domainFiltersat{{ .Values.domain }}(same value the hostnames use) and drop the now-unuseddomainNamefromvalues.yaml.2.
github-token-refresher-initJobs stuck the apps OutOfSync/MissingBoth copies (
argocd/andcrossplane-components/) usedhelm.sh/hook-delete-policy: hook-succeeded. ArgoCD deletes the Job on success but still tracks it as a desired resource → live state lacks it → app perpetuallyOutOfSync/Missingand retries the hook.Fix:
before-hook-creationso the completed Job persists.3.
apply-argocd-clusters-appflapped the argocd app every ~100sPlain Job with
ttlSecondsAfterFinished: 100and no hook annotation. Kubernetes auto-deletes it ~100s after completion while ArgoCD tracks it → theargocdApplication flapsSynced ↔ OutOfSync/Healthy ↔ Missingon a ~100s cycle.Fix: annotate as a
post-install,post-upgradehook withbefore-hook-creationand remove the TTL, so ArgoCD excludes it from the live-vs-desired diff.Notes
k1-civo/gitopsfor the livee2e-mgmt-cluster-k3mto unblock it.saas-rishi(the Helm v2 layout these clusters render from);mainuses a different (tokenized) layout and isn't affected by the same lines.