Repository navigation
Commit d090d29
IN LIST: retain short integer, floating-point, and fixed-size binary lists (#25187)
## Which issue does this PR close?
- Part of #19241.
## Rationale for this change
DataFusion currently rewrites short `IN` lists like `x IN (1, 2)` as `x
= 1 OR x = 2`. Evaluating `IN` directly can be faster.
This PR keeps the `IN` form for the types below, so short lists can use
DataFusion's existing optimized `IN` filters.
## What changes are included in this PR?
- Keep two- and three-value `IN` and `NOT IN` lists for `Int32`,
`Int64`, `UInt64`, `Float32`, `Float64`, and `FixedSizeBinary` columns
with widths of 1, 2, 4, 8, or 16 bytes. The list must contain constant
values of the column's type.
- Add benchmarks that use normal optimization to compare performance
before and after this change.
- Add optimizer and SQL logic tests covering list sizes, types, NULLs,
and non-constant list values.
## What is the testing strategy for this PR?
All 935 optimizer unit tests and all 533 SQL logic files passed,
including TPC-H. Formatting, Clippy, and lint checks passed. All 336
benchmark cases passed on both builds.
The benchmark bot's ARM
[two-value](#25187 (comment))
and
[three-value](#25187 (comment))
results show:
- Integer lists take about 32–55% less time when half the rows match or
none do. In the extreme case where every row matches the first value,
they can take up to about twice as long.
- Floating-point and fixed-size binary lists are faster in every tested
case, including NULLs and small batches.
<details>
<summary>Benchmark setup</summary>
Both builds use the same 336 cases. The
[baseline](31a333d)
contains `main` at `06aa131cda0` plus the same benchmark code as the
[PR](8aea24b).
Queries use normal optimization. Timings cover running the chosen
expressions; query planning and table setup happen before timing.
</details>
## Are there any user-facing changes?
`EXPLAIN` shows `IN` or `NOT IN` for these short lists in place of the
expanded `OR` or `AND` comparisons.
This performance PR aligns constant two- and three-value
`Float32`/`Float64` lists with existing behavior for 4+ values,
including the signed-zero bug (`+0.0` versus `-0.0`). That bug [predates
this optimization
series](#7378) and already
affects longer lists with constant or column values; fixing it is left
to #25186.
---------
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>1 parent a9fc673 commit d090d29
11 files changed
Lines changed: 784 additions & 40 deletions
File tree
- datafusion
- core
- benches
- optimizer/src/simplify_expressions
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
245 | 245 | | |
246 | 246 | | |
247 | 247 | | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
248 | 252 | | |
249 | 253 | | |
250 | 254 | | |
| |||
0 commit comments