[geoffreyclaude]

Fork-only experiment for comphead’s fixed-group suggestion.

Rationale for this change

DataFusion #26242 limits simultaneous sorting tasks but creates one task per run: an independently sorted piece of input that the final merge consumes.

Can a fixed number of larger tasks reduce this overhead without leaving sorting work unevenly distributed?

What changes are included in this PR?

• Divide each partition’s runs into at most per_partition_limit contiguous groups, with approximately equal estimated memory sizes. This limit is the smaller of the Tokio worker count and available CPU parallelism.
• Each task sorts its group sequentially and yields between runs. It occupies one place in the shared task limit until the whole group finishes.
• The merge receives runs in their original order, preserving equal-key ordering.

This removes per-run task creation and result reordering, while adding group balancing and resource ownership for whole groups.

What is the testing strategy for this PR?

Baseline: #26242 at 416c2d6.

Run on 22 October 2026: Apple M1 Max, four Tokio workers, TPC-H SF1 lineitem (6,001,215 rows), optimized release-nonlto builds with mimalloc. Twelve balanced rounds per workload, with separate warmups; every measured run is retained.

“Narrow” sorts three integer columns. “Wide” returns all 16 columns and sorts by four keys, including text. The output target is 8,192 rows. Preloaded cases exclude reading; Parquet cases include it. Sort timings exclude planning.

Median query execution times:

Workload	Baseline	Fixed groups
Narrow, preloaded, 32 rows	2494 ms	2317 ms
Narrow, preloaded, 256 rows	803 ms	776 ms
Wide, preloaded, 256 rows	747 ms	785 ms
Narrow, Parquet, 8,192 rows	396 ms	400 ms
Wide, Parquet, 8,192 rows	857 ms	952 ms
Wide, preloaded, 8,192 rows	538 ms	571 ms
Uneven work, 8,192 rows	359 ms	394 ms

Conclusion: keep the per-run design in apache#26242. Fixed groups gained about 6% on the fragmented 32-row narrow sort, but cost 12.8% on wide Parquet (95% interval: +8.3% to +17.4%), 5.8% on preloaded wide input, and 9.0% with uneven work. In the uneven case, first output moved from 51 to 81 ms: finished groups cannot take work from a slower group.

Observed unrelated-query delays improved substantially. On the preloaded wide sort, the probe maximum fell from 42.4 to 0.8 ms.

Follow-up traces explain the gain: merge startup converts sort keys from every run’s first batch into a comparison format. In the baseline, all four workers could enter these long conversion loops at once. Fixed groups stagger merge startup, leaving workers available for other queries.

Focused tests cover group ordering, permit lifetime, cancellation, reservation cleanup and spilling.

Percentages above are median paired changes, which can differ from ratios of the table medians. Probe figures are medians of each run’s maximum SELECT 1 delay; short runs provide only dozens of probes. Timings use an unbounded memory pool.

See the reproduction guide, all measurements, and summary with intervals.

Are there any user-facing changes?

Experimental internal scheduling; public configuration and output ordering are preserved. This draft is intended to support the design discussion.

This PR was prepared with AI assistance (GPT-6.1 Sol Ultra).