fix: preserve Spark errors for Parquet timestamp overflow - #5891
fix: preserve Spark errors for Parquet timestamp overflow#5891peterxcli wants to merge 2 commits into
Conversation
andygrove
left a comment
There was a problem hiding this comment.
This gives users a bare java.lang.ArithmeticException for a failure that happens inside the Parquet reader, but Spark wraps that one. On 4.x, FileScanRDD.hasNext routes the reader's exception through FileDataSourceV2.attachFilePath, and on 3.4 and 3.5 it goes through the NonFatal(e) branch at the end of nextIterator. Both land on QueryExecutionErrors.cannotReadFilesError, so what a Spark user actually sees is a SparkException with FAILED_READ_FILE naming the file, caused by ArithmeticException: long overflow. After this change Comet throws the ArithmeticException on its own with no SparkThrowable anywhere in the chain, so intercept[SparkException] and anything reading getCondition still diverge.
That is also the opposite of what the other scan-path errors in this converter do. ParquetSchemaConvert, ParquetMissingFieldIds and CannotReadFile each wrap in cannotReadFilesError in ShimSparkErrorConverter so the outer exception is a SparkException, and SparkErrorConverterSuite has a test asserting exactly that. The new test does not catch the difference because isLongOverflow searches the whole cause chain, so a missing wrapper is invisible to it.
Would it make sense to wrap this case the same way, using the taskFilePaths already threaded into convertToSparkException to supply the path? One wrinkle is that #5457 raises the same LongOverflow from a cast, where a bare ArithmeticException really is what Spark throws, so the scan case probably needs its own error type or a wrap on the scan side rather than a blanket change in the converter. CometTestBase.checkSparkError would hold the line on class parity if you want the test to cover it.
rich7420
left a comment
There was a problem hiding this comment.
@peterxcli +1,LGTM thanks for the patch
Which issue does this PR close?
Closes #5517.
Rationale for this change
Parquet TIMESTAMP_MILLIS overflow currently surfaces as a raw Arrow error. Spark reports a file-read
SparkExceptionnaming the file, caused byArithmeticException("long overflow"), even with ANSI disabled.What changes are included in this PR?
Emit a Parquet-specific overflow error and reconstruct Spark's file-read wrapper using the task's file paths. Keep
LongOverflowunwrapped for casts, consistent with #5457.How are these changes tested?
Regression tests compare exception class, error condition, SQLSTATE, file path, and arithmetic cause for positive and negative overflow, dictionary pages, nested timestamps, and both ANSI settings, including scans feeding native shuffle. The stronger assertion fails before this fix. Native tests and focused Spark 3.4, 3.5, and 4.1 tests pass, including filtered-scan controls and unwrapped cast errors.