Repository navigation
IN LIST: retain short integer, floating-point, and fixed-size binary lists - #25187
geoffreyclaude wants to merge 3 commits into
Conversation
0186b14 to
ef1ad44
Compare
ef1ad44 to
11b0a61
Compare
11b0a61 to
41aaaa1
Compare
41aaaa1 to
46e59a1
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #25187 +/- ##
=======================================
Coverage 82.77% 82.77%
=======================================
Files 1147 1147
Lines 450909 450980 +71
Branches 450909 450980 +71
=======================================
+ Hits 373252 373316 +64
- Misses 54934 54939 +5
- Partials 22723 22725 +2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
46e59a1 to
d0ec00e
Compare
04a830c to
499119b
Compare
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 499119b (499119b) to bec8193 diff Run configurationrun benchmark in_list_rewrite
env:
BENCH_FILTER: "list=2/"
baseline:
ref: "bec81934341766b273dab7012ed127015d946cc6"
changed:
ref: "499119b2edbdecf40ce98fea090f50be6499750b"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 499119b (499119b) to bec8193 diff Run configurationrun benchmark in_list_rewrite
env:
BENCH_FILTER: "list=3/"
baseline:
ref: "bec81934341766b273dab7012ed127015d946cc6"
changed:
ref: "499119b2edbdecf40ce98fea090f50be6499750b"Results will be posted here when complete File an issue against this benchmark runner |
202998f to
73d1e26
Compare
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 73d1e26 (73d1e26) to 3787a2b diff Run configurationrun benchmark in_list_rewrite
env:
BENCH_FILTER: "^in_list_rewrite/(float32|float64|fixed_size_binary_(1|2|4|8|16))/.*/list=2/"
baseline:
ref: "3787a2b324288c743fb764bdb7b893d794b58b8c"
changed:
ref: "73d1e2609476d47af3c4d410d148722c4842736f"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 73d1e26 (73d1e26) to 3787a2b diff Run configurationrun benchmark in_list_rewrite
env:
BENCH_FILTER: "^in_list_rewrite/(int32|int64|uint64|utf8|utf8view_inline)/"
baseline:
ref: "3787a2b324288c743fb764bdb7b893d794b58b8c"
changed:
ref: "73d1e2609476d47af3c4d410d148722c4842736f"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 73d1e26 (73d1e26) to 3787a2b diff Run configurationrun benchmark in_list_rewrite
env:
BENCH_FILTER: "^in_list_rewrite/(float32|float64|fixed_size_binary_(1|2|4|8|16))/.*/list=3/"
baseline:
ref: "3787a2b324288c743fb764bdb7b893d794b58b8c"
changed:
ref: "73d1e2609476d47af3c4d410d148722c4842736f"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 73d1e26 (73d1e26) to 3787a2b diff Run configurationrun benchmark in_list_rewrite
env:
BENCH_FILTER: "^in_list_rewrite/(int32|int64|uint64|utf8|utf8view_inline)/"
baseline:
ref: "3787a2b324288c743fb764bdb7b893d794b58b8c"
changed:
ref: "73d1e2609476d47af3c4d410d148722c4842736f"CPU Details (lscpu)Details
Resource Usagein_list_rewrite — base (merge-base)
in_list_rewrite — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 73d1e26 (73d1e26) to 3787a2b diff Run configurationrun benchmark in_list_rewrite
env:
BENCH_FILTER: "^in_list_rewrite/(float32|float64|fixed_size_binary_(1|2|4|8|16))/.*/list=2/"
baseline:
ref: "3787a2b324288c743fb764bdb7b893d794b58b8c"
changed:
ref: "73d1e2609476d47af3c4d410d148722c4842736f"CPU Details (lscpu)Details
Resource Usagein_list_rewrite — base (merge-base)
in_list_rewrite — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 73d1e26 (73d1e26) to 3787a2b diff Run configurationrun benchmark in_list_rewrite
env:
BENCH_FILTER: "^in_list_rewrite/(float32|float64|fixed_size_binary_(1|2|4|8|16))/.*/list=3/"
baseline:
ref: "3787a2b324288c743fb764bdb7b893d794b58b8c"
changed:
ref: "73d1e2609476d47af3c4d410d148722c4842736f"CPU Details (lscpu)Details
Resource Usagein_list_rewrite — base (merge-base)
in_list_rewrite — branch
File an issue against this benchmark runner |
73d1e26 to
61e15e2
Compare
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 61e15e2 (61e15e2) to 0c42744 diff Run configurationrun benchmark in_list_rewrite
env:
BENCH_FILTER: "^in_list_rewrite/.*/list=3$"
baseline:
ref: "0c427448bb90240023770c81ecbbe1e3bb9129d6"
changed:
ref: "61e15e2249467a07ecc592fdcb5d932beabac1d9"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 61e15e2 (61e15e2) to 0c42744 diff Run configurationrun benchmark in_list_rewrite
env:
BENCH_FILTER: "^in_list_rewrite/.*/list=2$"
baseline:
ref: "0c427448bb90240023770c81ecbbe1e3bb9129d6"
changed:
ref: "61e15e2249467a07ecc592fdcb5d932beabac1d9"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 61e15e2 (61e15e2) to 0c42744 diff Run configurationrun benchmark in_list_rewrite
env:
BENCH_FILTER: "^in_list_rewrite/.*/list=2$"
baseline:
ref: "0c427448bb90240023770c81ecbbe1e3bb9129d6"
changed:
ref: "61e15e2249467a07ecc592fdcb5d932beabac1d9"CPU Details (lscpu)Details
Resource Usagein_list_rewrite — base (merge-base)
in_list_rewrite — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 61e15e2 (61e15e2) to 0c42744 diff Run configurationrun benchmark in_list_rewrite
env:
BENCH_FILTER: "^in_list_rewrite/.*/list=3$"
baseline:
ref: "0c427448bb90240023770c81ecbbe1e3bb9129d6"
changed:
ref: "61e15e2249467a07ecc592fdcb5d932beabac1d9"CPU Details (lscpu)Details
Resource Usagein_list_rewrite — base (merge-base)
in_list_rewrite — branch
File an issue against this benchmark runner |
| // specific language governing permissions and limitations | ||
| // under the License. | ||
|
|
||
| //! Benchmarks short SQL `IN` lists using the default optimizer. Comparing the |
There was a problem hiding this comment.
These are pretty exhaustive benchmarks. They were useful to figure out which types to OR-rewrite and which to keep as IN LIST: I'm not sure there's value in keeping all of them though.
b3c8f98 to
8aea24b
Compare
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 8aea24b (8aea24b) to 31a333d diff Run configurationrun benchmark in_list_rewrite
env:
BENCH_FILTER: "^in_list_rewrite/.*/list=3$"
baseline:
ref: "31a333da6952047803dcd93b7a80e59ef056ce8e"
changed:
ref: "8aea24b85b1e16e0c0e13773935b130e67160853"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 8aea24b (8aea24b) to 31a333d diff Run configurationrun benchmark in_list_rewrite
env:
BENCH_FILTER: "^in_list_rewrite/.*/list=2$"
baseline:
ref: "31a333da6952047803dcd93b7a80e59ef056ce8e"
changed:
ref: "8aea24b85b1e16e0c0e13773935b130e67160853"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 8aea24b (8aea24b) to 31a333d diff Run configurationrun benchmark in_list_rewrite
env:
BENCH_FILTER: "^in_list_rewrite/.*/list=2$"
baseline:
ref: "31a333da6952047803dcd93b7a80e59ef056ce8e"
changed:
ref: "8aea24b85b1e16e0c0e13773935b130e67160853"CPU Details (lscpu)Details
Resource Usagein_list_rewrite — base (merge-base)
in_list_rewrite — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 8aea24b (8aea24b) to 31a333d diff Run configurationrun benchmark in_list_rewrite
env:
BENCH_FILTER: "^in_list_rewrite/.*/list=3$"
baseline:
ref: "31a333da6952047803dcd93b7a80e59ef056ce8e"
changed:
ref: "8aea24b85b1e16e0c0e13773935b130e67160853"CPU Details (lscpu)Details
Resource Usagein_list_rewrite — base (merge-base)
in_list_rewrite — branch
File an issue against this benchmark runner |
Which issue does this PR close?
Rationale for this change
DataFusion currently rewrites short
INlists likex IN (1, 2)asx = 1 OR x = 2. EvaluatingINdirectly can be faster.This PR keeps the
INform for the types below, so short lists can use DataFusion's existing optimizedINfilters.What changes are included in this PR?
INandNOT INlists forInt32,Int64,UInt64,Float32,Float64, andFixedSizeBinarycolumns with widths of 1, 2, 4, 8, or 16 bytes. The list must contain constant values of the column's type.What is the testing strategy for this PR?
All 935 optimizer unit tests and all 533 SQL logic files passed, including TPC-H. Formatting, Clippy, and lint checks passed. All 336 benchmark cases passed on both builds.
The benchmark bot's ARM two-value and three-value results show:
Benchmark setup
Both builds use the same 336 cases. The baseline contains
mainat06aa131cda0plus the same benchmark code as the PR.Queries use normal optimization. Timings cover running the chosen expressions; query planning and table setup happen before timing.
Are there any user-facing changes?
EXPLAINshowsINorNOT INfor these short lists in place of the expandedORorANDcomparisons.This performance PR aligns constant two- and three-value
Float32/Float64lists with existing behavior for 4+ values, including the signed-zero bug (+0.0versus-0.0). That bug predates this optimization series and already affects longer lists with constant or column values; fixing it is left to #25186.