fix(chainsaw): guard DaemonSet health checks against stale rollouts

[mikecook]

Summary

Guards six DaemonSet-based chainsaw health checks (gke-nccl-tcpxo x2, aws-efa, nvidia-dra-driver-gpu, nfd, slinky-topograph, nvsentinel) against a stale rollout passing as healthy.

Motivation / Context

numberReady alone can't distinguish a node still running a DaemonSet's previous revision (also Ready) from a fully current rollout — a node on the previous revision reports Ready too, so none of the six checks could tell a current rollout from a stale one. Adds an updatedNumberScheduled/observedGeneration guard to all of them.

Fixes: N/A
Related: #1222 (established the numberReady > 0 vacuous-pass rationale this follows)

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update
• Refactoring (no functional changes)
• Build/CI/tooling

Component(s) Affected

• CLI (cmd/aicr, pkg/cli)
• API server (cmd/aicrd, pkg/server)
• Recipe engine / data (pkg/recipe)
• Bundlers (pkg/bundler, pkg/component/*)
• Collectors / snapshotter (pkg/collector, pkg/snapshotter)
• Validator (pkg/validator)
• Core libraries (pkg/errors, pkg/k8s)
• Docs/examples (docs/, examples/)
• Other: recipes/checks/* health-check.yaml (chainsaw health checks), pkg/chainsaw test coverage

Implementation Notes

Each check's DaemonSet assertion gains two additional JMESPath conditions alongside the existing numberReady == desiredNumberScheduled: updatedNumberScheduled == desiredNumberScheduled and observedGeneration == metadata.generation. Both are omitempty status fields, so they're compared with the (field || 0) pattern rather than a bare equality, consistent with the existing desiredNumberScheduled > 0 vacuous-pass guard from #1222.

nvsentinel/health-check.yaml required new check entries rather than a small diff (hence the +46/-0), and pkg/chainsaw/nvsentinel_check_states_test.go was extended to cover the new assertions. Golden files (stock_render_golden.yaml, catalog_parity_golden.yaml) are regenerated output from the health-check.yaml changes — regenerate with make update-goldens if this drifts further before merge.

Testing

make qualify

All stages passed (test-coverage, lint, tuning-check, coverage-check, e2e, scan, license-check, api-diff). pkg/chainsaw package coverage: 81.9%. scan surfaced only pre-existing low/unknown-severity findings unrelated to this change (GO-2025-3547 low, GO-2026-5932 unknown) — no new highs.

Risk Assessment

• Low — Isolated change, well-tested, easy to revert

Rollout notes: N/A — tightens existing health-check assertions only; a currently-passing, fully-rolled-out DaemonSet is unaffected. A cluster mid-rollout will now correctly fail the check instead of a false pass, which is the intended behavior change.

Checklist

• Tests pass locally (make test with -race)
• Linter passes (make lint)
• I did not skip/disable tests to make CI green
• I added/updated tests for new functionality
• I updated docs if user-facing behavior changed
• Changes follow existing patterns in the codebase
• Commits are cryptographically signed (git commit -S)

[github-actions]

Recipe evidence check

No leaf overlays affected by this PR.

This gate is warning-only and never blocks merge.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: bb523328-4385-4569-8bde-218574a1378b

📥 Commits

Reviewing files that changed from the base of the PR and between b6b2420 and 32c4480.

📒 Files selected for processing (9)

• pkg/bundler/testdata/stock_render_golden.yaml
• pkg/chainsaw/nvsentinel_check_states_test.go
• pkg/recipe/testdata/catalog_parity_golden.yaml
• recipes/checks/aws-efa/health-check.yaml
• recipes/checks/gke-nccl-tcpxo/health-check.yaml
• recipes/checks/nfd/health-check.yaml
• recipes/checks/nvidia-dra-driver-gpu/health-check.yaml
• recipes/checks/nvsentinel/health-check.yaml
• recipes/checks/slinky-topograph/health-check.yaml

Included review availability: Your plan provides up to 12 included reviews per hour; 10 remain after this review.

---

📝 Walkthrough

Walkthrough

The pull request updates multiple DaemonSet health checks to require scheduled, ready, updated, and generation-synchronized rollouts. NV Sentinel tests add stale-rollout fixtures and failure cases for two DaemonSets. Generated stock-render and recipe catalog golden files contain updated overlay digests.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to 32c44

This change tightens DaemonSet health checks to reject stale rollouts while preserving healthy fully current rollouts; no actionable merge-blocking risk remains after normal checks and review.

Suggested reviewers: arangogutierrez

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely identifies the primary change: guarding Chainsaw DaemonSet health checks against stale rollouts.
Description check	✅ Passed	The description directly explains the stale-rollout issue, affected health checks, added conditions, tests, and intended behavior.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}