Skip to content

fix(gpu): force disable eBPF probes #3302

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
betterengineering wants to merge 2 commits into
base: main
Choose a base branch
from
Open

fix(gpu): force disable eBPF probes #3302

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
764b0d9
fix(gpu): force disable eBPF probes
betterengineering Jul 23, 2026
310273c
fix(gpu): disable eBPF probes on core agent too
betterengineering Jul 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions internal/controller/datadogagent/feature/gpu/envvar.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,3 +17,6 @@ const NVIDIAVisibleDevicesEnvVar = "NVIDIA_VISIBLE_DEVICES"

// DDPatchCgroupPermissionsEnvVar is the name of the system-probe gpu_monitoring module cgroup permissions patch knob
const DDPatchCgroupPermissionsEnvVar = "DD_GPU_MONITORING_CONFIGURE_CGROUP_PERMS"

// DDEnableEBPFProbesEnvVar is the name of the system-probe gpu_monitoring module eBPF probes enablement knob
const DDEnableEBPFProbesEnvVar = "DD_GPU_MONITORING_ENABLE_EBPF_PROBES"
15 changes: 15 additions & 0 deletions internal/controller/datadogagent/feature/gpu/feature.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,21 @@ func configureSystemProbe(managers feature.PodTemplateManagers) {
// add the env var to the core agent as well, to prevent config mismatches in runtime
managers.EnvVar().AddEnvVarToContainer(apicommon.CoreAgentContainerName, enableSPEnvVar)

// In privileged mode the eBPF probes are disabled, as GPU monitoring relies on
// the privileged host access rather than the eBPF probes.
disableEBPFProbesEnvVar := &corev1.EnvVar{
Name: DDEnableEBPFProbesEnvVar,
Value: "false",

@mbertrone mbertrone Jul 24, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion (non-blocking): this force-disables the probes, which makes the privilegedMode field doc stale. The marker at api/datadoghq/v2alpha1/datadogagent_types.go:721 still reads "PrivilegedMode enables GPU Probe module in System Probe" — accurate for the module, but the probes it implies are now off by default. That comment is the source for both generated docs (docs/configuration_public.md:235, docs/configuration.v2alpha1.md:136), so they drift with it. Update the marker to note the eBPF probes are disabled by default and privileged mode is retained for cgroup permission patching, then run make generate && make manifests so both generated docs move in lockstep.

}

// disable the eBPF probes in system-probe
managers.EnvVar().AddEnvVarToContainer(apicommon.SystemProbeContainerName, disableEBPFProbesEnvVar)

// add the env var to the core agent as well, so both containers agree that the
// eBPF probes are disabled and the core GPU check does not poll a system-probe
// module that has the probes turned off
managers.EnvVar().AddEnvVarToContainer(apicommon.CoreAgentContainerName, disableEBPFProbesEnvVar)

@mbertrone mbertrone Jul 24, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion (non-blocking): this hard-codes the probes off with no first-class opt-out, which is the right default given the GB300 crashes. Worth documenting the escape hatch that already exists: feature config runs before spec.override, and env merge is last-writer-wins by name (OverrideCurrentEnvVarMergeFunction in merger/envvars.go), so a user who still needs the probes can set DD_GPU_MONITORING_ENABLE_EBPF_PROBES=true via spec.override.nodeAgent.containers and it wins cleanly, with no duplicate env var. A line in the release note / docs saying probes are now disabled by default in privileged mode and can be re-enabled that way would save a support round-trip.


// annotations
managers.Annotation().AddAnnotation(common.SystemProbeAppArmorAnnotationKey, common.SystemProbeAppArmorAnnotationValue)

Expand Down
8 changes: 8 additions & 0 deletions internal/controller/datadogagent/feature/gpu/feature_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -193,6 +193,10 @@ func Test_GPUMonitoringFeature_Configure(t *testing.T) {
Name: DDEnableGPUProbeEnvVar,
Value: "true",
},
{
Name: DDEnableEBPFProbesEnvVar,
Value: "false",
},
{
Name: NVIDIAVisibleDevicesEnvVar,
Value: "all",
Expand All @@ -216,6 +220,10 @@ func Test_GPUMonitoringFeature_Configure(t *testing.T) {
Name: DDEnableGPUProbeEnvVar,
Value: "true",
},
{
Name: DDEnableEBPFProbesEnvVar,
Value: "false",
},
{
Name: common.DDSystemProbeSocket,
Value: common.DefaultSystemProbeSocketPath,
Expand Down
Loading