Skip to content

Job Monitor retry loops against Helix can take too long #17311

Description

@premun

The monitor resubmitted one work item only but Helix was giving 500s at that time.
The Helix submission kept retrying for 50 minutes until the job timed out.

Not sure if JobSender problem or JobMonitor one.

2026-08-11T09:30:46.5323531Z info: Monitoring Helix jobs for stage Test of build 1547565:
2026-08-11T09:30:46.5324214Z       https://dev.azure.com/dnceng-public/public/_build/results?buildId=1547565
2026-08-11T09:30:46.5325090Z info: 📖 Read more about what this monitor job does: https://github.com/dotnet/arcade/blob/main/src/Microsoft.DotNet.Helix/Sdk/Readme.md#helix-job-monitor-for-azure-devops
2026-08-11T09:30:46.7351534Z info: 🔁 Checking for failed Helix jobs to resubmit the failed work items...
2026-08-11T09:30:47.1892664Z info: 🔁 Resubmitting 1 work item(s) for job 'Windows_NT Build_Release - windows.11.amd64.client.open (1c3139e9-8f0c-41a4-818f-1b0c0ebf4a72)' (1 by Helix exit code, 0 by failed AzDO tests):
2026-08-11T09:30:47.1894050Z       - Microsoft.DotNet.Helix.Sdk.Tests.dll (Helix exit code -1)
2026-08-11T09:30:47.1922038Z info: Resubmitting 1 failed work item(s) for job Windows_NT Build_Release - windows.11.amd64.client.open (1c3139e9-8f0c-41a4-818f-1b0c0ebf4a72):
2026-08-11T09:30:47.1922913Z       - Microsoft.DotNet.Helix.Sdk.Tests.dll
2026-08-11T10:27:31.0026573Z ##[error]Helix Job Monitor terminated with an unhandled exception.
2026-08-11T10:27:31.0032478Z       Microsoft.DotNet.Helix.Client.RestApiException`1[Microsoft.DotNet.Helix.Client.Models.ApiError]: The response contained an invalid status code 500 Internal Server Error
2026-08-11T10:27:31.0032717Z       
2026-08-11T10:27:31.0033040Z       Body: {"Message":"An error occured.","ActivityId":"9318133c8437fc9b11d115123aacecf1"}
2026-08-11T10:27:31.0033424Z          at Microsoft.DotNet.Helix.JobMonitor.HelixService.RetryAsync[T](Func`1 action, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Helix/JobMonitor/Services/HelixService.cs:line 431
2026-08-11T10:27:31.0033832Z          at Microsoft.DotNet.Helix.JobMonitor.HelixService.ResubmitWorkItemsAsync(HelixJobInfo originalJob, IReadOnlyCollection`1 failedWorkItems, String targetStageAttempt, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Helix/JobMonitor/Services/HelixService.cs:line 342
2026-08-11T10:27:31.0034247Z          at Microsoft.DotNet.Helix.JobMonitor.JobMonitorRunner.ExecuteRetryPassAsync(CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Helix/JobMonitor/JobMonitorRunner.cs:line 204
2026-08-11T10:27:31.0034582Z          at Microsoft.DotNet.Helix.JobMonitor.JobMonitorRunner.RunAsync(CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Helix/JobMonitor/JobMonitorRunner.cs:line 89
2026-08-11T10:27:31.0034877Z          at Microsoft.DotNet.Helix.JobMonitor.Program.Main(String[] args) in /_/src/Microsoft.DotNet.Helix/JobMonitor/Program.cs:line 86
2026-08-11T10:27:31.0163069Z 
2026-08-11T10:27:31.0175554Z ##[error]Bash exited with code '1'.
2026-08-11T10:27:31.0229902Z ##[section]Finishing: Monitor Helix Jobs

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions