[CORE] Fix Delta 4.0 Spark 4.1 package build

[malinjawi]

What changes were proposed in this pull request?

This PR fixes the nightly package failure for the Spark 4.1 + Delta Velox build path.

Changes:

• Use Spark Logging plus explicit SparkMDC.of(...) calls in Delta 4.0 logging code so the code compiles across Spark 4.0, Spark 4.1, Delta 4.0, and Delta 4.1.
• Add Spark 4.1 compatibility shims for Delta 4.0 references to the old streaming CheckpointFileManager and MemoryStream packages.
• Use the Spark 4.1 Delta artifact, delta-spark_4.1_2.13:4.1.0, from the Spark 4.1 Maven profile. Delta 4.0.0 still references Spark TimeAdd, which is absent in Spark 4.1.
• Align Delta 4.0 test shims with Spark 4.1 / Delta 4.1 error classes for default-column and temp-view insert checks.
• Add Spark 4.1 -Pdelta test-compile coverage inside the regular spark-test-spark41 workflow before running the existing Spark 4.1 unit tests.

Why are the changes needed?

The nightly package build compiles Spark 4.1 with -Pdelta, but PR CI did not enable Delta on the Spark 4.1 Velox path. Spark 4.1 changed the logging MDC shape and moved streaming APIs used by Delta 4.0, so the original issue only surfaced in nightly packaging.

After enabling Spark 4.1 Delta coverage in PR CI, two more Spark 4.1 issues surfaced: Delta 4.0.0 references org.apache.spark.sql.catalyst.expressions.TimeAdd, which is not present in Spark 4.1, and the reused Delta 4.0 tests need Spark 4.1-specific error expectations. The Spark 4.1 profile now resolves Delta 4.1.0 using the Spark-specific Delta artifact, and the tests select the Spark 4.1 expectations using org.apache.spark.SPARK_VERSION.

How was this patch tested?

• ./build/mvn -ntp -pl shims/spark41 clean install -Pspark-4.1 -Pscala-2.13 -Pjava-21 -DskipTests -Dmaven.source.skip
• ./build/mvn -ntp -pl backends-velox -am clean install -Pspark-4.1 -Pscala-2.13 -Pjava-21 -Pbackends-velox -Pceleborn -Puniffle -Pdelta -DskipTests -Dmaven.source.skip
• ./build/mvn -ntp -pl backends-velox -am clean install -Pspark-4.0 -Pscala-2.13 -Pjava-21 -Pbackends-velox -Pceleborn -Puniffle -Piceberg -Phudi -Pdelta -Ppaimon -DskipTests -Dmaven.source.skip
• ./build/mvn -ntp -pl backends-velox -am clean test-compile -Pspark-4.1 -Pscala-2.13 -Pjava-21 -Pbackends-velox -Pceleborn -Puniffle -Pdelta -Ddelta.package.name=delta-spark_4.1 -Ddelta.version=4.1.0 -Ddelta.binary.version=40 -DskipTests -Dmaven.source.skip
• ./build/mvn -ntp -pl backends-velox -am clean test-compile -Pspark-4.1 -Pscala-2.13 -Pjava-21 -Pbackends-velox -Pceleborn -Puniffle -Pdelta -DskipTests -Dmaven.source.skip
• ./build/mvn -ntp -pl backends-velox -am clean test-compile -Pspark-4.0 -Pscala-2.13 -Pjava-21 -Pbackends-velox -Pceleborn -Puniffle -Pdelta -DskipTests -Dmaven.source.skip
• ./build/mvn -ntp -pl backends-velox -am clean test-compile -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pbackends-velox -Pspark-ut -Pdelta -DskipTests -Dmaven.source.skip
• ./build/mvn -ntp -q help:evaluate -Dexpression=delta.package.name -DforceStdout -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pdelta
• ./build/mvn -ntp -q help:evaluate -Dexpression=delta.version -DforceStdout -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pdelta
• zipgrep -n TimeAdd ~/.m2/repository/io/delta/delta-spark_4.1_2.13/4.1.0/delta-spark_4.1_2.13-4.1.0.jar
• git diff --check

[zhouyuan]

the fast-fail test is designed for AI tool
could we enable -Pdelta for Spark-4.1 test?
https://github.com/apache/gluten/blob/main/.github/workflows/velox_backend_x86.yml#L1378

[malinjawi]

Thanks @zhouyuan. Got it, I moved the coverage from the fast-build job to the regular Spark 4.1 Velox test command, so spark-test-spark41 now runs with -Pdelta.

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86