slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1233710	dotnet/aspnetcore	Test: Blazor E2E tests on Linux	Log	dotnet/aspnetcore#64742
1233658	dotnet/maui	Controls CollectionView	Log	dotnet/maui#33177
2862466	dotnet-runtime	Performance ios_scenarios iOSCoreCLR Interpreter iOSStripSymbols osx x64 perfiphone12mini net11.0	Log
1233488	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
1233381	dotnet/maui	Controls CollectionView	Log	dotnet/maui#33127
1233292	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#51934
2862246	dotnet-runtime	coreclr Pri1 Runtime Tests Run osx arm64 checked	Log
1233092	dotnet/maui	Controls Editor,Effects,FlyoutPage,Focus,Fonts,Frame,Gestures,GraphicsView	Log	dotnet/maui#33183
1232836	dotnet/maui	Controls ScrollView	Log	dotnet/maui#33173
1232841	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
1231402	dotnet/maui	Controls CollectionView	Log	dotnet/maui#32674
1232765	dotnet/maui	Controls Editor,Effects,FlyoutPage,Focus,Fonts,Frame,Gestures,GraphicsView	Log	dotnet/maui#33161
1232697	dotnet/maui	Controls Layout	Log	dotnet/maui#33157
1232261	dotnet/runtime	tvos-arm64 Release AllSubsets_Mono	Log
1232258	dotnet/maui	Controls Shell	Log	dotnet/maui#33168
1231816	dotnet/maui	Run DeviceTests iOS	Log	dotnet/maui#33004
1231815	dotnet/maui	Controls CollectionView	Log	dotnet/maui#33004
1228398	dotnet/maui	macOS Unit Tests	Log	dotnet/maui#33028
1231163	dotnet/runtime	iossimulator-arm64 Release AllSubsets_Mono_RuntimeTests	Log
1230918	dotnet/maui	Controls Editor,Effects,FlyoutPage,Focus,Fonts,Frame,Gestures,GraphicsView	Log	dotnet/maui#33152
1230448	dotnet/maui	Controls Editor,Effects,FlyoutPage,Focus,Fonts,Frame,Gestures,GraphicsView	Log	dotnet/maui#33144
1230265	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#122478
1229783	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#122495
1229772	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#122497
1229699	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#121305
1229650	dotnet/maui	Controls CollectionView	Log	dotnet/maui#32479
1229598	dotnet/maui	Controls Editor,Effects,FlyoutPage,Focus,Fonts,Frame,Gestures,GraphicsView	Log	dotnet/maui#33134
1229583	dotnet/maui	Controls Editor,Effects,FlyoutPage,Focus,Fonts,Frame,Gestures,GraphicsView	Log	dotnet/maui#33132
1229544	dotnet/xharness	OSX Release	Log	dotnet/xharness#1525
1229497	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#122388
1228385	dotnet/maui	Build macOS	Log	dotnet/maui#33122
1229441	dotnet/maui	Build macOS	Log	dotnet/maui#33100
1228371	dotnet/maui	Build macOS	Log	dotnet/maui#33120
1229448	dotnet/runtime	tvos-arm64 Release AllSubsets_Mono	Log
1229389	dotnet/runtime	osx-arm64 Release Libraries_CheckedCoreCLR	Log
1228245	dotnet/xharness	OSX Debug	Log	dotnet/xharness#1525
1229353	dotnet/maui	Controls Editor,Effects,FlyoutPage,Focus,Fonts,Frame,Gestures,GraphicsView	Log	dotnet/maui#33128
1229334	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#122407
1229270	dotnet/runtime	tvos-arm64 Release AllSubsets_Mono	Log	dotnet/runtime#122480
1229214	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#122478
1229093	dotnet/maui	Controls CollectionView	Log	dotnet/maui#33127
1228971	dotnet/runtime	Libraries Test Run checked coreclr osx arm64 Debug	Log	dotnet/runtime#122469
1229011	maestro-auth-test/helix-queue-insights-test	HelixTests	Log
1228892	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#122211
1228455	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
1228804	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
1228658	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT_Smoke	Log	dotnet/runtime#122460
1228731	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#122454
1228677	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#122463
1228338	dotnet/runtime	tvos-arm64 Release AllSubsets_Mono_Smoke	Log	dotnet/runtime#119575
1228190	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT_Smoke	Log	dotnet/runtime#122454
1228170	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT_Smoke	Log	dotnet/runtime#122414
1228146	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT_Smoke	Log	dotnet/runtime#122381
1228130	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#122451
1228102	dotnet/runtime	tvos-arm64 Release AllSubsets_Mono_Smoke	Log	dotnet/runtime#122448
1228099	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#122023
1228022	dotnet/runtime	tvos-arm64 Release AllSubsets_Mono_Smoke	Log	dotnet/runtime#122409
1228013	dotnet/runtime	tvos-arm64 Release AllSubsets_Mono_Smoke	Log	dotnet/runtime#122436
1227972	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT_Smoke	Log	dotnet/runtime#121672
1227964	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT_Smoke	Log	dotnet/runtime#122441
1227902	dotnet/maui	Build macOS	Log	dotnet/maui#33076
1227912	dotnet/runtime	tvos-arm64 Release AllSubsets_Mono_Smoke	Log	dotnet/runtime#122439
1227876	dotnet/runtime	tvos-arm64 Release AllSubsets_Mono_Smoke	Log	dotnet/runtime#122388
1227836	dotnet/maui	RunOnAndroid macOS	Log	dotnet/maui#33084
1227837	dotnet/maui	Controls Editor,Effects,FlyoutPage,Focus,Fonts,Frame,Gestures,GraphicsView	Log	dotnet/maui#33084
1227826	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono_RuntimeTests	Log
1227677	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
1227523	dotnet/runtime	tvos-arm64 Release AllSubsets_Mono_Smoke	Log	dotnet/runtime#119432
1227488	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#122425
1227478	dotnet/runtime	tvos-arm64 Release AllSubsets_Mono_Smoke	Log	dotnet/runtime#121513
1227451	dotnet/runtime	tvos-arm64 Release AllSubsets_Mono_Smoke	Log	dotnet/runtime#122409
1227466	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT_Smoke	Log	dotnet/runtime#122333
1227379	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono	Log	dotnet/runtime#122373
1227380	dotnet/runtime	tvos-arm64 Release AllSubsets_Mono	Log	dotnet/runtime#122371
1227287	dotnet/runtime	tvos-arm64 Release AllSubsets_Mono_Smoke	Log	dotnet/runtime#122415
1227275	dotnet/runtime	tvos-arm64 Release AllSubsets_Mono_Smoke	Log	dotnet/runtime#122032
1227268	dotnet/runtime	tvos-arm64 Release AllSubsets_Mono_Smoke	Log	dotnet/runtime#122230
1227184	dotnet/runtime	tvos-arm64 Release AllSubsets_Mono_Smoke	Log	dotnet/runtime#122410
1227168	dotnet/runtime	tvos-arm64 Release AllSubsets_Mono_Smoke	Log	dotnet/runtime#122408
2858445	dotnet-runtime	Performance ios_scenarios iOSNativeAOT NativeAOT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1227080	dotnet/runtime	tvos-arm64 Release AllSubsets_Mono	Log
1226770	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#122234
1226726	dotnet/maui	Build macOS	Log	dotnet/maui#33099
1226710	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#122375
1226608	dotnet/maui	MultiProject macOS	Log	dotnet/maui#33100
1226625	dotnet/runtime	tvos-arm64 Release AllSubsets_Mono_Smoke	Log	dotnet/runtime#122308
1226534	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#122386
1226467	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#122381
1226452	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#119575
1226449	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#121956
2858114	dotnet-performance	Performance powershell windows Win11 x64 Viper main	Log
1226362	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#122380
1226228	dotnet/runtime	Libraries Test Run release coreclr osx arm64 Release	Log
1226274	dotnet/runtime	osx-x64 Release NativeAOT_Libs	Log	dotnet/runtime#122227
1226265	dotnet/runtime	iossimulator-arm64 Release AllSubsets_Mono_RuntimeTests	Log
1226245	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#122227
1226183	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
2857926	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
2857862	dotnet-sdk	TestBuild: macOS (x64)	Log	#55975
1226020	dotnet/runtime	Libraries Test Run release coreclr osx arm64 Debug	Log	dotnet/runtime#122375
Displaying 100 of 473 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
8	92	473

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.