Improve Performance (Polars 2.0 Call to Action) #26169

Is your feature request related to a problem or challenge?

@ritchie46 posted a blog https://pola.rs/posts/release-polars-2/ recently that provides benchmark numbers for Polars 2.0 that make Datafusion look quite bad:

I suspect it is telling that

Describe the solution you'd like

I would like the performance of DataFusion to be better (obviously)

Describe alternatives you've considered

My strong suspicion is that the difference is largely caused by the fact that TPC-H and TPC-DS require good join ordering for high performance, something that DataFusion has been historically weak at

I also think this problem is also likely due to the same thing:

Polars and both DuckDB versions completed all queries. DataFusion timed out on TPC-DS q72 (and once on q67) and ran out of memory on TPC-H q18 on c7a.4xlarge; those queries are excluded from the results above for all engines.

Another corroborating piece of evidence is that the post does not mention ClickBench https://benchmark.clickhouse.com/ where we have focused a huge amount of effort on benchmarking and the difference is much less pronounced. See for example the results here:

Additional context

No response

Willingness to contribute

None