Skip to content

[VL] Add KeyGroupedPartitioning support to columnar shuffle#12084

Open
minni31 wants to merge 1 commit into
apache:mainfrom
minni31:oss/velox-keygrouped-partitioning
Open

[VL] Add KeyGroupedPartitioning support to columnar shuffle#12084
minni31 wants to merge 1 commit into
apache:mainfrom
minni31:oss/velox-keygrouped-partitioning

Conversation

@minni31
Copy link
Copy Markdown

@minni31 minni31 commented May 13, 2026

CONTEXT

KeyGroupedPartitioning is a Spark partitioning scheme used by V2 data source connectors (e.g., Iceberg, Paimon) where data is partitioned by specific key expressions with known unique partition values. Currently, Gluten's columnar shuffle exchange does not handle this partitioning type, causing a fallback to vanilla Spark for any query involving V2 sources with key-grouped partitioning.

WHAT

Adds KeyGroupedPartitioning support to the columnar shuffle exchange path in the Velox backend. The implementation reuses the existing JVM-side partition ID computation pattern (same mechanism as RangePartitioning):

  • Adds KeyGroupedPartitioning to the validation whitelist in ColumnarShuffleExchangeExecBase, allowing the columnar shuffle to accept this partitioning type.
  • Constructs a KeyGroupedPartitioner from the partitioning's uniquePartitionValues, mapping each partition key to its index.
  • Computes partition IDs on the JVM side by evaluating partition key expressions against each row (via BindReferences) and looking up the result in the KeyGroupedPartitioner. The pid column is prepended to each batch so the native shuffle writer can read it directly.
  • Reuses RangePartitioningShortName for the native partitioning descriptor since both Range and KeyGrouped use the same JVM-side pid prepend pattern — the native shuffle writer reads the prepended column rather than computing partition IDs natively.
  • Each key extraction allocates a fresh array per row and converts to an immutable ArraySeq to avoid aliasing issues with mutable array reuse.

Tests

Suite Tests Status
VeloxShufflePartitioningSuite 22 tests covering hash, range, round-robin, single partitioning, null semantics, data types, boundary cases, and KeyGrouped unit tests (key extraction, partitioner construction, end-to-end partition lookup) Local pass

Note: End-to-end KeyGroupedPartitioning tests require V2 data source connectors (Iceberg/Paimon) which are not available in this test module. The KeyGrouped unit tests validate key extraction, KeyGroupedPartitioner construction, and the full key-extraction-to-partition-lookup flow.

@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@github-actions github-actions Bot added CORE works for Gluten Core VELOX labels May 13, 2026
@minni31 minni31 force-pushed the oss/velox-keygrouped-partitioning branch from 02ddcb4 to 92ac0bc Compare May 13, 2026 08:39
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CORE works for Gluten Core VELOX

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant