[Performance]: Preserve early filtering for partition-state OR, large PREFIX_IN, and mixed ranges filters

[XuPeng-SH]

Is there an existing issue for performance?

• I have checked the existing issues.

Environment

• Version or commit-id: 7821d21c72
• Branch: optimize_filter_0711
• OS: Linux
• Scope: partition-state/workspace reads and ranges block-filter compilation

Details of Performance

Three confirmed paths preserve correctness but lose available early filtering and can scan substantially more in-memory rows, blocks, or object metadata than necessary.

1. Partition-state OR/disjunct filters

   • BasePKFilter.Disjuncts makes NewMemPKFilter invalid because the partition-state primary-key iterator accepts only one atomic predicate.
   • Reads therefore fall back to NewRowsIter and scan all visible in-memory rows even when every OR branch is individually primary-key-filterable.

2. Large PREFIX_IN in workspace reads

   • MemPKFilter.FilterVector intentionally returns without filtering when PREFIX_IN contains more than four prefixes.
   • This avoids repeated linear matching, but turns a selective prefix set into a full workspace scan.

3. All-or-nothing ranges compilation

   • readutil.CompileFilterExprs marks the entire filter list as non-compilable when any one expression lacks a fast compiler implementation.
   • The supported predicates then also lose object/block-level fast filtering, causing avoidable metadata loads and block scans.

These are performance gaps, not correctness bugs: residual SQL filters still produce correct query results.

Expected Behavior

• OR/disjunct primary-key predicates should use a safe union-capable partition-state iterator, or an equivalent bounded batch filter, whenever all branches are supported.
• Large PREFIX_IN sets should use a scalable representation (for example, sorted prefixes with binary search/trie/range pruning) instead of disabling filtering.
• Ranges compilation should retain all independently compilable fast filters and route only unsupported predicates through the slow evaluator.

Steps to Reproduce

1. Create a table with a primary key and leave selective rows in the transaction workspace/partition state.
2. Execute a predicate that produces BasePKFilter.Disjuncts, such as pk = 1 OR pk = 1000000; observe that the partition-state path cannot use the primary-key iterator.
3. Execute PREFIX_IN with more than four prefixes against workspace rows; observe that MemPKFilter.FilterVector skips early filtering.
4. Construct a BlockFilterList containing both a compiler-supported predicate and one zonemappable but unsupported predicate; observe CompileFilterExprs return canCompile=false for the whole list.

Proposed Direction and Acceptance Criteria

• Extend the partition-state iterator contract to represent OR safely; do not pass an unsupported synthetic op into existing single-range iterators.
• Add scalable PREFIX_IN filtering with bounded memory and explicit tests for 0, 1, 4, 5, and large prefix counts.
• Split ranges filters into fast-compilable and slow-evaluated subsets while preserving AND semantics and fail-open behavior on compiler/filter errors.
• Add observability/tests for partition-state rows scanned, blocks selected, metadata/object loads, and fast-vs-slow compilation ratio.
• Verify correctness for NULLs, empty lists, overlapping prefixes, mixed OR branches, malformed filter state, and cancellation/error paths.

Additional information

This issue is the follow-up to the composite-key block-filter work in #21740. The current implementation deliberately keeps the three paths conservative to avoid false negatives; the goal is to remove the performance fallback without weakening that safety property.

[Ariznawlll]

已在官方最新 main 完成白盒回归验证，结论是：#25588 描述的三项性能降级当前仍然存在，尚未修复。

验证环境

• 验证日期：2026-07-13
• 仓库：matrixorigin/matrixone
• 分支：main
• commit：2865ed6cc8a39727b8b274bb52b88fe441d0b30c
• commit title：fix(plan): harden folded IN vectors for composite keys (#25641)
• 平台：macOS arm64 / Apple M3
• 说明：该 main commit 已包含 fix: strengthen filter pushdown and PK pruning #25596 的合并提交 de6438665。

本轮增加了三个临时白盒复现测试。测试断言记录的是当前的降级行为，因此测试 PASS 表示问题被稳定复现，不表示问题已修复。

1. Partition-state OR/disjunct 退化

构造等价于以下 SQL 条件的 PK filter：

pk = 1 OR pk = 1000000

验证结果：

1. planner/ConstructBasePKFilter 可以生成两个 BasePKFilter.Disjuncts；仓库现有 TestConstructBasePKFilterWithOr 也明确覆盖了 eq OR eq 生成 disjuncts 的真实路径。
2. 同一组 disjuncts 交给 block reader 时，可以生成有效的联合 SortedSearchFunc 和 UnSortedSearchFunc。
3. 同一组 disjuncts 交给 NewMemPKFilter 时，返回的 filter 为 Valid=false。
4. 因此 partition-state datasource 会进入 NewRowsIter 分支，不能使用 PK iterator 对 OR 分支做选择性读取。

该路径不是只在手工构造 filter 时存在；实际 planner 的 eq OR eq 会产生 Disjuncts。

2. Workspace 大 PREFIX_IN 阈值退化

构造 8192 个全部不匹配的 workspace PK 值，分别使用不同数量的 PREFIX_IN 前缀调用 MemPKFilter.FilterVector。

前缀数量	skipMask 提前过滤行数
0	8192
1	8192
4	8192
5	0
64	0
1024	0

从 4 个前缀增加到 5 个前缀后，提前过滤从 8192 行突变为 0 行；5、64、1024 三种规模都会完整保留全部不匹配行，交给后续执行层继续处理。

Benchmark（每项 20x，重复 5 次）也得到同样结果：

prefixes-4:  350~406 us/op, 8192 early-skipped/op
prefixes-5:  259~284 us/op,    0 early-skipped/op
prefixes-64: 260~279 us/op,    0 early-skipped/op

5/64 项在这个微基准里表面更快，是因为函数直接放弃了过滤；这不是查询性能提升，后续 workspace reader 会承担放大的扫描和残余表达式处理。

3. Mixed ranges filter 全有或全无

构造以下 AND 条件：

pk = 999999 AND abs(v) = 7

其中 pk = 999999 是可快速编译的 PK 条件，abs(v) = 7 作为不能由 ranges fast compiler 独立编译的残余条件。

验证结果：

• 单独编译 pk = 999999：
  • canCompile=true
  • fast/load/object/block filter 均不为空。
• 使用 CompileFilterExprs([pk = 999999, abs(v) = 7])：
  • canCompile=false
  • fast/load/object/block/seek 全部为 nil。
• 调换条件顺序为 [abs(v) = 7, pk = 999999] 后结果相同。

说明一个不可编译的表达式仍会导致已经可以编译的 PK fast filter 一并丢失，无法保留部分 object/block pruning。

稳定性和回归结果

聚焦复现测试：

go test ./pkg/vm/engine/readutil -run '^TestIssue25588' -count=100
ok github.com/matrixorigin/matrixone/pkg/vm/engine/readutil

Race 测试：

go test -race ./pkg/vm/engine/readutil -run '^TestIssue25588' -count=20
ok github.com/matrixorigin/matrixone/pkg/vm/engine/readutil

全包回归：

go test ./pkg/vm/engine/readutil -count=1
ok github.com/matrixorigin/matrixone/pkg/vm/engine/readutil

未发现数据竞争或结果正确性错误；确认的是三条 fast-path 的扫描放大问题。

最终判断

当前官方 main@2865ed6cc 上：

• partition-state OR/disjunct 仍会失去内存 PK iterator；
• PREFIX_IN > 4 仍会完全跳过 workspace 提前过滤；
• mixed ranges 仍会因一个不可编译表达式丢弃整组 fast filter。

因此建议保持 issue Open。修复后的验收条件应分别为：OR 使用安全的 union/批量选择性 iterator；5+ PREFIX_IN 正确生成 skip mask；mixed ranges 保留可独立编译的 fast filters，并由残余 SQL filter 继续保证最终语义。

[Ariznawlll]

@XuPeng-SH 鹏哥这个问题再看下