Helix Job Monitor cannot complete when work items are permanently stranded in 'Waiting'

[mmitche]

Summary

The Helix Job Monitor (src/Microsoft.DotNet.Helix/JobMonitor) can never complete when a Helix job it is monitoring contains work items that are permanently stranded in the Waiting state (created but never dispatched to a machine). Because the monitor's job discovery is scoped by build + stage (not by pipeline attempt), retrying the monitor job re-discovers the same stranded work and re-drives the full timeout every time.

This is not a code bug — the monitor is faithfully implementing the spec — but the design decision makes it impossible for the monitor to ever reach a terminal (completed) state in this scenario. Filing so we can decide how the monitor should handle abandoned/never-scheduled work items.

Observed scenario

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1513096 (dotnet/runtime PR #129690)
Job: Monitor Helix Jobs (record b0da1d21-03df-59a7-d49e-ac6d68c91909)

Every monitor attempt logs the same stable status and then times out at the 535‑minute (8h55m) budget:

ℹ️ Status: 80 processed / 80 completed / 3 running / 0 waiting jobs
             6810 processed / 6810 completed / 83 running / 6 waiting work items
...
##[error]Helix Job Monitor timed out after 535 minute(s) (08:55:00). 4 Helix job(s) were unfinished or unprocessed:

The blocking Helix job is 62175fbc-d087-4e2b-b648-a2605eba4334 (windows-x64 Checked NativeAOT, queue windows.10.amd64.open.rt), created 2026-07-16T19:52 UTC. Live Helix state:

• totalWorkItems: 9, passed: 5, inProgress: 4
• The 4 in-progress items are all state: "Waiting", exitCode: -1, machineName: null — never scheduled, days after creation:
  • Loader
  • LoaderClassloaderGenerics
  • LoaderClassloaderRegressions
  • async

Stranded work-item API URIs:

https://helix.dot.net/api/2019-06-17/jobs/62175fbc-d087-4e2b-b648-a2605eba4334/workitems/Loader
https://helix.dot.net/api/2019-06-17/jobs/62175fbc-d087-4e2b-b648-a2605eba4334/workitems/LoaderClassloaderGenerics
https://helix.dot.net/api/2019-06-17/jobs/62175fbc-d087-4e2b-b648-a2605eba4334/workitems/LoaderClassloaderRegressions
https://helix.dot.net/api/2019-06-17/jobs/62175fbc-d087-4e2b-b648-a2605eba4334/workitems/async

Root cause

A Helix job is only counted complete when it is finished/failed, or (fallback) when every expected work item has a terminal exit code (JobMonitorRunner.AreAllWorkItemsTerminalAsync):

workItems.Count >= job.InitialWorkItemCount.Value
    && workItems.All(wi => wi.ExitCode.HasValue);

The stranded Waiting items never get an ExitCode, so the job never completes, and termination (all monitored AzDO jobs complete and all scoped Helix jobs complete — Design.md §5.4/§5.5) is never reached.

Discovery is scoped by build + stage, not by pipeline attempt (Design.md §2.1; HelixService.GetJobsForBuildAsync filters on the Helix source string plus the BuildId property, then System.StageName). An AzDO "retry" is a new attempt of the same BuildId, so each retry re-discovers the identical stranded job and re-runs the full 8h55m timeout. This cross-attempt scoping is intentional (crash/timeout resilience — re-attach to prior attempts' jobs to upload results and resubmit failures, Design.md §2.2/§2.6), so per-attempt scoping is not a free fix.

These items are also not eligible for the monitor's own resubmission path: retry only resubmits failed items of a completed latest incarnation; a still-Waiting incarnation is skipped (Design.md §2.3 rules 3 & 6).

Note: the same job reports InitialWorkItemCount=6 but totalWorkItems=9, so the initial-count heuristic is also stale here — though the All(ExitCode.HasValue) half is the actual blocker.

The gap

The monitor has no notion of a permanently stranded / abandoned work item. It cannot distinguish "legitimately still Waiting (queued, will run soon)" from "abandoned in the queue, will never run." As a result the only exit for this state is the global timeout, and every retry repeats it.

Possible directions (for discussion)

1. Stranded-Waiting detection / fail-fast (keeps build+stage scope, preserves resilience): treat a work item that has been Waiting with no machine assignment for longer than some threshold (absolute age, or age since monitor start) as abandoned. Surface it as a hard failure with actionable output instead of silently waiting out the full timeout, so retries don't each burn ~9 hours.
2. Per-attempt scoping for discovery: only monitor jobs submitted by the current stage attempt. Simpler mental model, but trades away the §2.2/§2.6 crash-resilience / upload-recovery guarantees.
3. Escalate to Helix service side: investigate why work items get stranded in Waiting on windows.10.amd64.open.rt and whether Helix should itself terminate/abandon never-scheduled work items after some SLA.

cc @dotnet/dnceng

[mmitche]

Investigation: why the stranded Waiting work items weren''t cancelled

Conclusion: this is not a bug in the monitor''s cancellation logic — the monitor does request cancellation of the stranded job on every timeout. The reason the items persist is a Helix-side (service) behavior.

The monitor''s cancel path does run, and does target the stranded job

1. Program.cs sets an in-process timeoutCts = --max-wait-minutes, which helix-job-monitor.yml wires to timeoutInMinutes - 5. So the tool''s own timeout fires ~5 minutes before Azure DevOps would force-kill the job, guaranteeing the graceful shutdown path runs. The ##[error]Helix Job Monitor timed out after 535 minute(s) line we see in the logs is emitted by StatusReporter.ReportTimeout(), which only executes on that graceful path — so we know it ran.
2. Immediately after ReportTimeout(), RunAsync''s catch (OperationCanceledException) calls CancelInFlightHelixJobsAsync with a fresh 30s cancellation budget.
3. CancelInFlightHelixJobsAsync selects the latest, non-completed, non-processed associated jobs:

   MonitorState.GetLatestHelixJobAttempts(_state.SnapshotAssociatedJobs())
       .Where(j => !j.IsCompleted && !_state.IsHelixJobProcessed(j.JobName))

   The stranded job 62175fbc-... (status running, 5 passed / 4 Waiting) is not completed and not processed, and nothing supersedes it, so it is selected and Job.CancelAsync (POST /api/jobs/{job}/cancel) is issued for it on every timeout.

So cancel-candidate selection is correct; the monitor is not silently skipping these jobs.

Why the Waiting items still never terminate (Helix-side)

Helix job cancellation is cooperative / eventual. A work item in Waiting with machineName: null was never dispatched to a machine. The terminal exit code / cancellation transition for a work item is driven by the Helix client running on the machine that picks it up. For an item that is never scheduled — here on a starved/removed queue (windows.10.amd64.open.rt) — there is no client to observe the cancel token or assign an exit code, so the item stays Waiting / exitCode -1 indefinitely. The job-level cancel marks the job but does not synthesize terminal exit codes for never-dispatched work items. (Repeat cancels just return 304 Already cancelled.)

Because these items are effectively un-cancellable from the monitor''s side, and discovery was scoped by build + stage across attempts, every monitor retry re-discovered the same stranded job and re-ran the full ~9h timeout — the loop described in this issue.

Resolution direction taken

Rather than trying to make the monitor force-terminate Helix-side stranded work (a Helix service gap), we''re taking Possible direction #2 — per-attempt scoping: the monitor now only tracks Helix jobs stamped with its own System.StageAttempt. On a stage retry, the new attempt''s submitters resubmit all work fresh, and the previous attempt''s stranded/failed work is abandoned instead of re-discovered — so a retried monitor completes normally instead of re-burning the timeout.

Escalating the Helix-side behavior (direction #3 — why items strand in Waiting on windows.10.amd64.open.rt, and whether Helix should abandon never-scheduled work after an SLA) is still worth a separate follow-up with the Helix service team.