Skip to content

fix: preserve Spark errors for Parquet timestamp overflow - #5891

Open
peterxcli wants to merge 2 commits into
apache:mainfrom
peterxcli:fix/parquet-timestamp-overflow-error
Open

fix: preserve Spark errors for Parquet timestamp overflow#5891
peterxcli wants to merge 2 commits into
apache:mainfrom
peterxcli:fix/parquet-timestamp-overflow-error

Conversation

@peterxcli

@peterxcli peterxcli commented Sep 13, 2026

Copy link
Copy Markdown
Member

Which issue does this PR close?

Closes #5517.

Rationale for this change

Parquet TIMESTAMP_MILLIS overflow currently surfaces as a raw Arrow error. Spark reports a file-read SparkException naming the file, caused by ArithmeticException("long overflow"), even with ANSI disabled.

What changes are included in this PR?

Emit a Parquet-specific overflow error and reconstruct Spark's file-read wrapper using the task's file paths. Keep LongOverflow unwrapped for casts, consistent with #5457.

How are these changes tested?

Regression tests compare exception class, error condition, SQLSTATE, file path, and arithmetic cause for positive and negative overflow, dictionary pages, nested timestamps, and both ANSI settings, including scans feeding native shuffle. The stronger assertion fails before this fix. Native tests and focused Spark 3.4, 3.5, and 4.1 tests pass, including filtered-scan controls and unwrapped cast errors.

@github-actions github-actions Bot added bug Something isn't working area:scan Parquet scan / data reading labels Sep 13, 2026

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This gives users a bare java.lang.ArithmeticException for a failure that happens inside the Parquet reader, but Spark wraps that one. On 4.x, FileScanRDD.hasNext routes the reader's exception through FileDataSourceV2.attachFilePath, and on 3.4 and 3.5 it goes through the NonFatal(e) branch at the end of nextIterator. Both land on QueryExecutionErrors.cannotReadFilesError, so what a Spark user actually sees is a SparkException with FAILED_READ_FILE naming the file, caused by ArithmeticException: long overflow. After this change Comet throws the ArithmeticException on its own with no SparkThrowable anywhere in the chain, so intercept[SparkException] and anything reading getCondition still diverge.

That is also the opposite of what the other scan-path errors in this converter do. ParquetSchemaConvert, ParquetMissingFieldIds and CannotReadFile each wrap in cannotReadFilesError in ShimSparkErrorConverter so the outer exception is a SparkException, and SparkErrorConverterSuite has a test asserting exactly that. The new test does not catch the difference because isLongOverflow searches the whole cause chain, so a missing wrapper is invisible to it.

Would it make sense to wrap this case the same way, using the taskFilePaths already threaded into convertToSparkException to supply the path? One wrinkle is that #5457 raises the same LongOverflow from a cast, where a bare ArithmeticException really is what Spark throws, so the scan case probably needs its own error type or a wrap on the scan side rather than a blanket change in the converter. CometTestBase.checkSparkError would hold the line on class parity if you want the test to cover it.

@peterxcli
peterxcli requested a review from andygrove September 13, 2026 16:38

@rich7420 rich7420 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@peterxcli +1,LGTM thanks for the patch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:scan Parquet scan / data reading bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Surface Parquet TIMESTAMP_MILLIS overflow as a Spark-faithful exception instead of a raw Arrow error

3 participants