ci: next set of jobs to move from the PR tier to the merge queue tier

[andygrove]

What is the problem the feature request solves?

#5843 split CI into a PR tier and a merge-queue tier. This issue proposes the next set of tier changes, based on the last 30 days of CI data (every CI run on a pull request from Aug 12 to Sep 11, 1,288 runs that actually executed, plus 168 push-to-main runs of the full pipeline used as the flake baseline).

"Unique catches" below means PR runs where the job failed while the Linux build, lint and the Spark 4.1 Linux test profile were all green, i.e. no cheaper job would have flagged the PR.

Cost and signal of the current PR tier

Job	Runner hrs/day	PR fail	Main fail	Unique catches / month	Wall clock
Spark SQL 4.1, sql_core-1/2/3 (+ build)	~96	8.5%	4.5%	33	100 min
Spark SQL 4.1, sql_hive-1/2/3	~65	1.4%	1.3%	7	110 min
Iceberg 1.11	84	6.4%	2.6%	17	100 min
Linux tests, Spark 3.4 profile	46	11.9%	6.0%	25	60 min
Linux tests, Spark 3.5 profile	44	11.3%	6.0%	20	60 min
Linux tests, Spark 4.0 profile	48	10.5%	5.4%	15	60 min
Linux tests, Spark 4.2 profile	45	10.0%	4.8%	19	60 min
Linux tests, Spark 4.1 profile	48	9.9%	4.8%	58	60 min
Linux build, lint, rust-test, TPC verify	58	15.3%	5.8%	120	55 min
Delta gate, PyArrow UDF, CodeQL, preflight	~9	low	0%	33	under 20 min

The PR tier costs roughly 540 runner-hours per day. The five Linux test profiles co-fail with each other 70 to 88% of the time. 60% of Iceberg 1.11 failures die at its "Build Comet" step, which the Linux build already catches.

The queue tier is not healthy enough to take more jobs yet

On known-good main, the jobs that are now queue-only went red 32% of the time over the month and 40% in September. With max_entries_to_build: 2 and a 2.5 hour pipeline that eviction rate will stall the queue. Two culprits:

• macOS: red on main 11% of the time (8% in the week of Sep 8). 79 of its 89 unique PR failures are the [scans] suite crashing with SIGSEGV in hdfsThreadDestructor (Sporadic SIGSEGV in macOS [scans] workflow in CI #5023, proposed fix fix: prevent libhdfs thread destructor use-after-free #5036, not yet merged). A Sep 9 comment on Sporadic SIGSEGV in macOS [scans] workflow in CI #5023 reports the same crash on ubuntu-24.04.
• Iceberg 1.8/1.9/1.10: red on main 24% of the time in September. About half of that was Iceberg native scan fails on a table partitioned by an unknown transform (TestForwardCompatibility CI failure) #5758 (fixed Sep 8); the rest is artifact-store 403/409 errors, Maven connection resets, and a Scala 2.13 Iceberg test flake.

Without macOS and the old Iceberg versions, the queue tier's red rate on main is 5 to 9%.

Describe the potential solution

In order:

1. Stabilize the queue tier first.

   • Land fix: prevent libhdfs thread destructor use-after-free #5036 (libhdfs thread destructor use-after-free). Until the crash rate on main is near zero, make macOS [scans] label-only rather than queue-tier.
   • Track the Iceberg 1.8/1.9/1.10 red rate on main after Iceberg native scan fails on a table partitioned by an unknown transform (TestForwardCompatibility CI failure) #5758 and the artifact retry changes (ci: add a Required Checks aggregator job so main can have a required status check #5842). If it stays above a few percent, move those three to label-only as well.

2. Move Spark SQL Tests (Spark 4.1) sql_hive-1/2/3 to queue-only. 65 runner-hours per day for 7 unique catches a month, and it is the 110 minute long pole. Keeps sql_core-1/2/3 and catalyst on PRs, which catch 33 real failures a month across 33 different branches. PR wall clock drops to about 100 minutes. This needs a hive on/off input on spark_sql_test_reusable.yml and a corresponding POLICY entry in dev/ci/compute-changes.py.

3. Move Iceberg Spark SQL Tests (Iceberg 1.11) to queue-only (opt in on PRs with run-iceberg-tests). 84 runner-hours per day for 17 unique catches, which become roughly one queue eviction every two days.

4. Optionally move the Spark 4.0 and 4.2 Linux test profiles to queue-only. 93 runner-hours per day for 34 unique catches. Keep 3.4 (Scala 2.12 and the oldest Spark, the most unique catches of the non-4.1 profiles) and 4.1 on PRs. Same mechanism as ci: move the Spark 3.4/3.5/4.0, Iceberg, macOS and benchmark suites behind a merge queue #5843 for the macOS build: a profile filter input on pr_build_linux.yml driven by POLICY.

Steps 2 and 3 save about 150 runner-hours per day (28% of PR spend) for about 24 escapes a month. Adding step 4 brings the saving to about 240 runner-hours per day (45%).

Not recommended: dropping the Spark SQL sql_core modules from PRs. They are the best-value heavy job after the Linux build.

Additional context

• The merge queue ruleset is active but there were zero merge_group runs at the time of the analysis (ci: move the Spark 3.4/3.5/4.0, Iceberg, macOS and benchmark suites behind a merge queue #5843 and deps: bump to datafusion 55.1.0 #5865 merged just before it took effect), so queue-tier behavior above is inferred from push-to-main runs.
• Unique-catch counts include some flakes; the "Main fail" column is the calibration for that. On main, a flaky failure hits all Linux profiles 80 to 100% of the time, so the per-profile unique catches are mostly real version-specific failures.
• The .github/workflows/README.md tier table and diagram need updating with each step.

[andygrove]

cc @blaginin @comphead @mbutrovich