Generate a native ORCA plan for a replicated CTE in scalar subqueries#384
Open
Alena0704 wants to merge 2 commits into
Open
Generate a native ORCA plan for a replicated CTE in scalar subqueries#384Alena0704 wants to merge 2 commits into
Alena0704 wants to merge 2 commits into
Conversation
Contributor
Author
|
I decided to create about new pull request because we fixed that problem before with fall back walker and now I am developing approach making orca builds the query plan for cross slice scalar subqueries without breaking the semantics of checking cross slice pattern for other components with replicated tables (the regression test proved it). FYI, the previous PR to fix this topic and discussion #370 |
Alena0704
force-pushed
the
bugfix-cross-slice-shared-scan-cte
branch
from
82f0b49 to
26c2063
Compare
Alena0704
changed the title
Alena0704
changed the title
Alena0704
force-pushed
the
bugfix-cross-slice-shared-scan-cte
branch
from
26c2063 to
8eb4869
Compare
Alena0704
force-pushed
the
bugfix-cross-slice-shared-scan-cte
branch
2 times, most recently
from
ac442fe to
92486db
Compare
Alena0704
changed the title
Alena0704
force-pushed
the
bugfix-cross-slice-shared-scan-cte
branch
from
92486db to
433247e
Compare
…ueries Root cause is a blind spot in the #375 slice walker: CollectCTESlices delimits slices only at Motion nodes. A CTE over a DISTRIBUTED REPLICATED table referenced from correlated scalar subqueries is decorrelated by ORCA into CPhysicalCorrelated*NLJoin whose inner side becomes an executor SubPlan running in its own slice -- but there is no Motion at that boundary, so the walker placed the Consumer on the same slice as the Producer. The cross-slice check (prod->sliceId != cons->sliceId) never fired, no fallback happened, and the ShareInputScan writer hung forever in shareinput_writer_waitdone() waiting for DONE acks from reader slices that never run. Teach the walker that the inner (subquery) side of a correlated NL join is a slice boundary too, mirroring the Motion rule. The replicated Consumer in the SubPlan then gets a distinct slice id, the existing check fires, and ORCA falls back to the Postgres optimizer.
A CTE over a DISTRIBUTED REPLICATED table referenced from several scalar subqueries put ORCA's SharedScan Producer and Consumer on different slices, and that cross-slice SharedScan hung. We used to detect the shape before DXL translation and fall back to the Postgres planner; now ORCA handles the repairable case natively. The fix is in apply_shareinput_xslice (src/backend/cdb/cdbmutate.c): a cross-slice Consumer inside a SubPlan over a replicated CTE gets a local copy of the Producer's subtree (Materialize + base Scan) with a fresh share_id in its own slice. The data is on every segment, so the local copy is equivalent. Consumers of the same CTE in one slice share the copy; orphaned original Producers are dropped. This only works when the CTE body is a single replicated base scan. So the pre-DXL check (CUtils) falls back unless the Consumer is inside a correlated NL join's SubPlan AND the Producer body is a single base scan. A UNION ALL (Append) body, or a broadcast/duplicate-hazard join (greengage 51fe92e) where the Consumer is not in a SubPlan, still falls back.
Alena0704
force-pushed
the
bugfix-cross-slice-shared-scan-cte
branch
from
433247e to
9e126af
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a CTE over a DISTRIBUTED REPLICATED table is referenced from several
scalar subqueries, ORCA puts the SharedScan Producer and Consumer on
different slices. That cross-slice SharedScan used to hang.
Until now we just avoided the hang: FHasCrossSliceReplicatedCTEConsumer
detected this shape before DXL translation and fell back to the Postgres
planner. This change lets ORCA handle the scalar-subquery case natively,
so no fallback is needed: the replicated CTE is materialized once and
shared by all references inside ORCA's own plan.
The fix is in apply_shareinput_xslice (src/backend/cdb/cdbmutate.c). When
a cross-slice Consumer is found inside a SubPlan and the CTE source is a
replicated table, the Consumer gets its own local copy of the Producer's
subtree (Materialize + base Scan) with a fresh share_id, marked
SHARE_MATERIAL and placed in the Consumer's slice. The table is
replicated, so every segment already has the full data and the local copy
is equivalent -- this removes the cross-slice coordination. Other
Consumers of the same CTE in the same slice reuse this copy (tracked by
(orig_share_id, motId) -> new_share_id), so the CTE is materialized once
and read by all references. cleanup_orphaned_producers then drops the
original Producers that are no longer used. The reuse map and consumer
counts live in new ApplyShareInputContext fields in
src/include/nodes/relation.h.
The pre-DXL fallback check (CUtils::FHasCrossSliceReplicatedCTEConsumer)
was too broad -- it fired for every cross-slice replicated CTE Consumer.
Narrow it to the join case only: a CTE Consumer under a
duplicate-hazard / broadcast Motion (greengage 51fe92e), which still
can't be handled natively. The scalar-subquery case no longer matches
here and reaches the new materialization path instead.