Helix Job Monitor seems to be waiting on something but all tests have passed

[AndyAyersMS]

https://dev.azure.com/dnceng-public/public/_build/results?buildId=1541153&view=logs&s=6884a131-87da-5381-61f3-d7acc3b91d76&j=078edc60-2a90-5618-e72d-426a045b11f0

I need to get this PR merged so may bypass this... I'll wait an hour or two post approval, in case somebody's around to take a look.

FYI @premun @mmitche

[AndyAyersMS]

PR is dotnet/runtime#131493 in case that helps.

[AndyAyersMS]

Second PR of mine is also hanging the same way: dotnet/runtime#131841

[AndyAyersMS]

In #131493 it eventually timed out, after 346 minutes.

[AndyAyersMS]

Looks like there was maybe a test failure, it just wasn't obvious what it was from the github status page; everything looked like it was passing.

[matouskozak]

Just noticed on https://github.com/dotnet/runtime/pull/131765/checks?check_run_id=92581980902, the helix monitor missed a test failure (System.Net.Sockets.Tests.SendReceive_Eap.TcpReceiveSendGetsCanceledByDispose) and only lists the wasm one. Not sure if related to the timeouts reported by Andy (can open a separate issue if necessary).

[mmitche]

@matouskozak Did it get that test failure on the second run through?

[matouskozak]

@matouskozak Did it get that test failure on the second run through?

I tried re-running the helix monitor CI and the result was the same.

[premun]

@mmitche I checked the builds linked by Andy and there technically is nothing wrong with the execution, just the test uploads took some sweet time to complete.

[18:25] info: 0 test results for job 'coreclr  Pri0 Runtime Tests Run  osx arm64 checked - osx.26.arm64.open (ad9cbf75-8a09-4cd4-827c-9d11583caf79)' processed.
[18:43] info: 126730 test results for job 'Libraries Test Run checked coreclr linux x64 Release - azurelinux.3.amd64.open.rt (bdc072a9-8f03-467b-ac40-437b48b4ea36)' processed.
[19:15] info: 126627 test results for job 'Libraries Test Run checked coreclr linux arm64 Debug - azurelinux.3.arm64.open (11ebf8b7-2406-4d69-8940-9d1262dcffcf)' processed.
[19:45] info: 127209 test results for job 'Libraries Test Run checked coreclr linux_musl x64 Debug - azurelinux.3.amd64.open.rt (a0d34e94-cc7e-46cd-8e75-997ade5b71d7)' processed.
[20:12] info: 75 test results for job 'browser-wasm linux Release CoreCLR_WasmBuildTests - azurelinux.3.amd64.open.rt (334e0fef-b2d5-4f3b-9e38-288cab977d66)' processed.
[20:13] info: 152 test results for job 'browser-wasm linux Release WasmBuildTests - azurelinux.3.amd64.open.rt (17de7e72-e53a-42af-9176-3ad67d3c572b)' processed.
[20:14] info: 152 test results for job 'browser-wasm linux Release WasmBuildTests - azurelinux.3.amd64.open.rt (9810e8c9-eaa9-4b5b-bfa7-f0ce6626702d)' processed.
[20:15] info: 127078 test results for job 'Libraries Test Run checked coreclr linux_musl arm Release - azurelinux.3.arm64.open (07acd6e7-ccd3-4333-bedd-f34cb83c32d7)' processed.
[20:42] info: 152 test results for job 'browser-wasm windows Release WasmBuildTests - windows.server2025.amd64.open.rt (2e895506-6c16-40b9-954a-8ead90fea6bb)' processed.
[20:43] info: 152 test results for job 'browser-wasm windows Release WasmBuildTests - windows.server2025.amd64.open.rt (fed105ba-c7f3-4769-b478-c0e4beae907d)' processed.
[20:43] info: 75 test results for job 'browser-wasm windows Release CoreCLR_WasmBuildTests - windows.server2025.amd64.open.rt (a400e226-bf82-4a81-8340-e4d60b8a5eab)' processed.
[20:46] info: 130102 test results for job 'Libraries Test Run checked coreclr windows x86 Debug - windows.10.amd64.open.rt (3237e077-0821-495b-8042-b96f9d68a3a0)' processed.
[21:05] info: 79428 test results for job 'browser-wasm linux Release LibraryTestsCoreCLR - azurelinux.3.amd64.open.rt (49aebffe-448a-4733-b1cf-ec21d74c5994)' processed.
[21:38] info: 130210 test results for job 'Libraries Test Run checked coreclr windows x64 Release - windows.10.amd64.open.rt (a4d4400f-0cad-4403-9ab5-8ca0e3685044)' processed.
[22:07] info: 525 test results for job 'osx-arm64 release Installer_Build_And_Test - osx.26.arm64.open (0172bde9-4850-4937-a513-0b0e8bdcf5d1)' processed.
[22:07] info: 124256 test results for job 'Libraries Test Run checked coreclr osx arm64 Debug - osx.26.arm64.open (d50e052a-c1c8-4d98-bb7f-0f916be1c43c)' processed.

I don't know if this is because we adhere to the headers that AzDO throttles us through or there is some other kind of starvation (like the semaphore over the test upload threads) but this is crazy slow.

[premun]

@matouskozak I think it does not explicitly list every failed test. The WASM one was a failed work item (returning non-zero).
I will check if in case of all work items succeeding, it would still fail because of that test (it should and I think it does).

[matouskozak]

@matouskozak I think it does not explicitly list every failed test. The WASM one was a failed work item (returning non-zero). I will check if in case of all work items succeeding, it would still fail because of that test (it should and I think it does).

If it doesn't list all the tests, where should we look to get a list of failures, the azdo test tab? Also if it doesn't list the failure and I trigger re-run of helix monitor, does it re-run the not-listed failure also or just the listed ones? Sorry if I missed any memo sent earlier explaining this.

[premun]

where should we look to get a list of failures, the azdo test tab?

Yes. Or the Build Analysis check page. This technically didn't change to how it was previously.

does it re-run the not-listed failure also or just the listed ones?

All failures, both work item with failed tests or failed work items. I will check why it stopped logging which work items have failed tests.