Skip to content

Commit d090d29

Browse files
IN LIST: retain short integer, floating-point, and fixed-size binary lists (#25187)
## Which issue does this PR close? - Part of #19241. ## Rationale for this change DataFusion currently rewrites short `IN` lists like `x IN (1, 2)` as `x = 1 OR x = 2`. Evaluating `IN` directly can be faster. This PR keeps the `IN` form for the types below, so short lists can use DataFusion's existing optimized `IN` filters. ## What changes are included in this PR? - Keep two- and three-value `IN` and `NOT IN` lists for `Int32`, `Int64`, `UInt64`, `Float32`, `Float64`, and `FixedSizeBinary` columns with widths of 1, 2, 4, 8, or 16 bytes. The list must contain constant values of the column's type. - Add benchmarks that use normal optimization to compare performance before and after this change. - Add optimizer and SQL logic tests covering list sizes, types, NULLs, and non-constant list values. ## What is the testing strategy for this PR? All 935 optimizer unit tests and all 533 SQL logic files passed, including TPC-H. Formatting, Clippy, and lint checks passed. All 336 benchmark cases passed on both builds. The benchmark bot's ARM [two-value](#25187 (comment)) and [three-value](#25187 (comment)) results show: - Integer lists take about 32–55% less time when half the rows match or none do. In the extreme case where every row matches the first value, they can take up to about twice as long. - Floating-point and fixed-size binary lists are faster in every tested case, including NULLs and small batches. <details> <summary>Benchmark setup</summary> Both builds use the same 336 cases. The [baseline](31a333d) contains `main` at `06aa131cda0` plus the same benchmark code as the [PR](8aea24b). Queries use normal optimization. Timings cover running the chosen expressions; query planning and table setup happen before timing. </details> ## Are there any user-facing changes? `EXPLAIN` shows `IN` or `NOT IN` for these short lists in place of the expanded `OR` or `AND` comparisons. This performance PR aligns constant two- and three-value `Float32`/`Float64` lists with existing behavior for 4+ values, including the signed-zero bug (`+0.0` versus `-0.0`). That bug [predates this optimization series](#7378) and already affects longer lists with constant or column values; fixing it is left to #25186. --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
1 parent a9fc673 commit d090d29

11 files changed

Lines changed: 784 additions & 40 deletions

File tree

‎datafusion/core/Cargo.toml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -245,6 +245,10 @@ name = "math_query_sql"
245245
harness = false
246246
name = "filter_query_sql"
247247

248+
[[bench]]
249+
harness = false
250+
name = "in_list_rewrite"
251+
248252
[[bench]]
249253
harness = false
250254
name = "struct_query_sql"

0 commit comments

Comments
 (0)