[AMORO-3973] Support spark3.4 for mixed format #4013
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
module:mixed-spark
turboFei
reviewed
Dec 15, 2025
| val schema = createSchema(transformClause.colTypeList) | ||
| val replacedSchema = CharVarcharUtils.replaceCharVarcharWithStringInSchema(schema) | ||
| (DataTypeUtils.toAttributes(replacedSchema), false) | ||
| (schema.toAttributes, false) |
Member
There was a problem hiding this comment.
should be replacedSchema.toAttributes?
Contributor
Author
There was a problem hiding this comment.
I fix the code by deleting 'replacedSchema'
| } | ||
|
|
||
| def visitCommonSelectQueryClausePlan( | ||
| private def visitCommonSelectQueryClausePlan( |
| <include>org.apache.iceberg:iceberg-orc</include> | ||
| <include>org.apache.iceberg:iceberg-parquet</include> | ||
| <include>org.apache.iceberg:iceberg-aws</include> | ||
| <include>org.apache.iceberg:iceberg-aliyun</include> |
Member
There was a problem hiding this comment.
is it necessary for spark-3.5 as well?
zhangwl9
force-pushed
the
AMORO-add-Spark3.4-for-mixedFormat-dev
branch
from
e2d8b8a to
c9082d4
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #4013 +/- ##
============================================
+ Coverage 22.12% 22.85% +0.72%
- Complexity 2461 2531 +70
============================================
Files 445 449 +4
Lines 40897 41048 +151
Branches 5767 5784 +17
============================================
+ Hits 9050 9382 +332
+ Misses 31089 30857 -232
- Partials 758 809 +51
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
zhangwl9
force-pushed
the
AMORO-add-Spark3.4-for-mixedFormat-dev
branch
from
c9082d4 to
c549de1
Compare
turboFei
reviewed
Dec 25, 2025
| <artifactId>scala-compiler</artifactId> | ||
| <version>${scala.version}</version> | ||
| <scope>provided</scope> | ||
| </dependency> |
Member
There was a problem hiding this comment.
why remove it?
If needed, you can remove it in separate PR for all spark-3.5, spark-3.4, spark-3.3
Member
There was a problem hiding this comment.
do not remove the <scope>provided</scope> for spark-3.4, please keep it consistent for spark-3.5, spark-3.4, spark-3.3, and change it in separate PR if necessary.
Contributor
Author
There was a problem hiding this comment.
As shown in the following picture,references to scala-compile and scala-compiler appear duplicated on lines 95–167 and 268–280; one instance requires removal.I will fixup it in another pr
zhangwl9
force-pushed
the
AMORO-add-Spark3.4-for-mixedFormat-dev
branch
4 times, most recently
from
e5fcedd to
ffdaeb7
Compare
added 4 commits
December 30, 2025 14:25
� Please enter the commit message for your changes. Lines starting � with '�' will be ignored, and an empty message aborts the commit. � � interactive rebase in progress; onto dd42830 � Last commands done (2 commands done): � pick 699f315 # Copy the code form spark3.5 � pick 701d7d5 # Optimize code � Next commands to do (2 remaining commands): � pick e2d8b8a # Optimize code � pick e5fcedd # Maintain consistency in the references to scala-compiler and scala-library across Spark 3.5, Spark 3.4, and Spark 3.3 � You are currently rebasing branch 'AMORO-add-Spark3.4-for-mixedFormat-dev' on 'dd428301a'. � � Changes to be committed: � renamed: amoro-format-mixed/amoro-mixed-spark/v3.4/amoro-mixed-spark-3.5/pom.xml -> amoro-format-mixed/amoro-mixed-spark/v3.4/amoro-mixed-spark-3.4/pom.xml � modified: amoro-format-mixed/amoro-mixed-spark/v3.4/amoro-mixed-spark-3.4/src/main/gen-antlr/org/apache/amoro/spark/sql/parser/MixedFormatSqlExtendLexer.java � modified: amoro-format-mixed/amoro-mixed-spark/v3.4/amoro-mixed-spark-3.4/src/main/gen-antlr/org/apache/amoro/spark/sql/parser/MixedFormatSqlExtendParser.java � renamed: amoro-format-mixed/amoro-mixed-spark/v3.4/amoro-mixed-spark-3.5/src/main/java/org/apache/amoro/spark/Spark35Adapter.java -> amoro-format-mixed/amoro-mixed-spark/v3.4/amoro-mixed-spark-3.4/src/main/java/org/apache/amoro/spark/Spark34Adapter.java � modified: amoro-format-mixed/amoro-mixed-spark/v3.4/amoro-mixed-spark-3.4/src/main/java/org/apache/amoro/spark/util/ExpressionHelper.java � modified: amoro-format-mixed/amoro-mixed-spark/v3.4/amoro-mixed-spark-3.4/src/main/scala/org/apache/amoro/spark/sql/catalyst/analysis/QueryWithConstraintCheck.scala � modified: amoro-format-mixed/amoro-mixed-spark/v3.4/amoro-mixed-spark-3.4/src/main/scala/org/apache/amoro/spark/sql/catalyst/analysis/RewriteMixedFormatMergeIntoTable.scala � modified: amoro-format-mixed/amoro-mixed-spark/v3.4/amoro-mixed-spark-3.4/src/main/scala/org/apache/amoro/spark/sql/catalyst/optimize/RewriteAppendMixedFormatTable.scala � modified: amoro-format-mixed/amoro-mixed-spark/v3.4/amoro-mixed-spark-3.4/src/main/scala/org/apache/amoro/spark/sql/catalyst/optimize/RewriteDeleteFromMixedFormatTable.scala � modified: amoro-format-mixed/amoro-mixed-spark/v3.4/amoro-mixed-spark-3.4/src/main/scala/org/apache/amoro/spark/sql/catalyst/optimize/RewriteUpdateMixedFormatTable.scala � modified: amoro-format-mixed/amoro-mixed-spark/v3.4/amoro-mixed-spark-3.4/src/main/scala/org/apache/amoro/spark/sql/catalyst/plans/UnresolvedMergeIntoMixedFormatTable.scala � renamed: amoro-format-mixed/amoro-mixed-spark/v3.4/amoro-mixed-spark-3.4/src/main/scala/org/apache/spark/sql/amoro/catalyst/MixedFormatSpark35Helper.scala -> amoro-format-mixed/amoro-mixed-spark/v3.4/amoro-mixed-spark-3.4/src/main/scala/org/apache/spark/sql/amoro/catalyst/MixedFormatSpark34Helper.scala � modified: amoro-format-mixed/amoro-mixed-spark/v3.4/amoro-mixed-spark-3.4/src/main/scala/org/apache/spark/sql/amoro/parser/MixedFormatSqlExtendAstBuilder.scala � modified: amoro-format-mixed/amoro-mixed-spark/v3.4/amoro-mixed-spark-3.4/src/main/scala/org/apache/spark/sql/catalyst/analysis/MixedFormatAssignmentAlignmentSupport.scala � modified: amoro-format-mixed/amoro-mixed-spark/v3.4/amoro-mixed-spark-3.4/src/test/java/org/apache/amoro/spark/test/suites/sql/TestMergeIntoSQL.java � modified: amoro-format-mixed/amoro-mixed-spark/v3.4/amoro-mixed-spark-3.4/src/test/java/org/apache/amoro/spark/test/suites/ut/sql/parser/TestSqlExtendParser.java � renamed: amoro-format-mixed/amoro-mixed-spark/v3.4/amoro-mixed-spark-runtime-3.5/pom.xml -> amoro-format-mixed/amoro-mixed-spark/v3.4/amoro-mixed-spark-runtime-3.4/pom.xml � modified: pom.xml �
…brary across Spark 3.5, Spark 3.4, and Spark 3.3
zhangwl9
force-pushed
the
AMORO-add-Spark3.4-for-mixedFormat-dev
branch
from
8b19bc4 to
9dffec2
Compare
Contributor
Author
|
@xxubai Would you be able to review this pr when you have a moment? I'd be most grateful. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are the changes needed?
Close #3973.
Brief change log
How was this patch tested?
Add some test cases that check the changes thoroughly including negative and positive cases if possible
Add screenshots for manual tests if appropriate
Run test locally before making a pull request
Documentation