fix: unify checked arithmetic and overflow semantics

[ck89119]

What type of PR is this?

• API-change
• BUG
• Improvement
• Documentation
• Feature
• Test and CI
• Code Refactoring

Which issue(s) this PR fixes:

issue #25754

What this PR does / why we need it:

Unifies checked arithmetic and overflow semantics across the SQL arithmetic execution kernels to match MySQL 8.4 strict-mode behavior (ERROR 1690), as required by the subtask of #25705.

Integer arithmetic

• Merges the 24 per-type integer overflow helpers (addInt8WithOverflowCheck ... mulUint64WithOverflowCheck) into 6 generic checked operations constrained by signedInt / unsignedInt. Error messages (type name and (%d op %d) format) are byte-for-byte identical to the old helpers.
• Fixes the signed-subtraction check, which previously required v1 > 0 and therefore missed the v1 - MinInt wraparound (e.g. 0 - (-9223372036854775808) silently wrapped; MySQL raises ERROR 1690).

Float arithmetic

• Adds overflow checks for float32/float64 -, *, /: finite operands producing Inf now raise out-of-range, consistent with + which already had the check.

DIV

• Float operands: the quotient is computed in float64 and truncated; any quotient with magnitude >= 2^63 (or NaN) raises out-of-range. Previously int64(v1/v2) silently truncated, so SELECT 1e308 DIV 1 returned garbage instead of erroring.
• MinInt64 DIV 1 and MinInt64 DIV -1 now raise out-of-range. MySQL computes DIV on unsigned magnitudes, so any quotient with magnitude 2^63 errors even when the signed result would fit (verified against MySQL 8.4.8).

Decimal

• Maps ErrInvalidInput from decimal kernels to ErrOutOfRange at the SQL arithmetic execution boundary.

Infrastructure

• specialTemplateForDivFunction now accepts error-returning kernels.
• Division-by-zero semantics (SQL-mode-controlled NULL vs. error) are unchanged and remain distinct from overflow errors.

Tests

All boundary assertions were validated against a local MySQL 8.4.8 instance:

• arithmetic_int_overflow_test.go: boundary matrix for all 8 integer types (+, -, *), including 0 - MinInt and MinInt * -1.
• arithmetic_float_overflow_test.go: float overflow helpers and decimal error mapping.
• arithmetic_overflow_boundary_test.go: DIV int64/uint64/float boundaries, MOD boundaries (MinInt64 MOD -1 = 0, matching MySQL), and div/mod template const/vector/null branch coverage.

Known deviations documented in test comments (out of scope here):

• MaxUint64 DIV 1: MySQL returns an UNSIGNED BIGINT; MO's DIV result type is fixed at signed int64, so it errors. Result type inference belongs to [Subtask]: add numeric resolver and prepared expression context #25751.
• MySQL converts float DIV operands through DECIMAL (~16 significant digits) before dividing; MO truncates in the double domain, so the low digits of extreme values can differ. Per-row decimal conversion is explicitly disallowed by the issue.

🤖 Generated with Claude Code

[XuPeng-SH]

There is one blocking semantic gap: the NULL/division-by-zero fix is limited to integer DIV, while the same row-eligibility contract is still violated by float /, float DIV, MOD, and decimal //DIV/MOD.

In specialTemplateForDivFunction, the vector/constant branch checks v2 == 0 before merging the dividend's NULLs (baseTemplate.go:1704-1718). specialTemplateForModFunction does the same (baseTemplate.go:1447-1458), and the decimal vec/const kernels return ErrDivByZero before consulting rsnull (for example arith_decimal_fast.go:893-900 and 4895-4902). Consequently, in strict DML mode, an all-NULL dividend vector over a constant zero incorrectly errors instead of producing NULL.

I reproduced this deterministically at the PR head:

• float64 /, DIV, and MOD: [NULL] op const(0) all returned division by zero;
• Decimal64 /, DIV, and MOD: an all-NULL bitmap with a constant-zero divisor all returned division by zero;
• an independent MySQL 8.0.44 strict-DML check inserted NULL successfully for all six corresponding DOUBLE/DECIMAL expressions.

The newly added TestIntegerDivNullDividendByZeroStrictMode does not cover the failing shape: its divisor is a normal vector, and it exercises only integer DIV.

Please enforce the underlying rule before any error-producing arithmetic: only rows that are selected and have two non-NULL operands may raise overflow or division-by-zero. Then cover /, DIV, and MOD across integer/unsigned/float/Decimal64/128/256; const/const, const/vector, vector/const, vector/vector; all-NULL and mixed-NULL dividends; partial selection masks; and strict/non-strict modes. A shared early "no evaluable rows" check is preferable to more per-type special cases.

Other review gates passed locally: the full function package, focused -race tests, go build, go vet, and git diff --check. I found no separate concurrency, ownership, or unbounded-growth issue, and no material hot-path allocation regression.

[gouhongshen]

Codex automated review

Strict-mode NULL propagation remains broken for constant-zero divisors outside the newly fixed integer-DIV loops.

P1 - Check row eligibility before raising a constant-zero error (pkg/sql/plan/function/baseTemplate.go:1709)

For a vector dividend and non-NULL constant zero divisor, this branch calls checkDivisionByZeroBehavior before merging the dividend null bitmap (the merge is only at line 1717). Thus, in strict INSERT/UPDATE mode, [NULL] / 0 and [NULL] DIV 0 for FLOAT/DOUBLE return division-by-zero instead of NULL. The same premature constant-zero pattern remains in specialTemplateForModFunction (line 1451) and decimal vec/const kernels (for example arith_decimal_fast.go:895), affecting /, DIV, and MOD for DECIMAL as well. Only selected rows with two non-NULL operands may raise this error; centralize that check and add strict all-NULL/mixed-NULL vector/const coverage.

[gouhongshen]

Codex automated review

Found one blocking masked-row overflow regression and one batch-reuse NULL-bitmap regression. Focused tests were blocked because cgo/libmo.dylib is missing; the worktree is unchanged.

P1 - Masked vector/constant float overflow is still evaluated (pkg/sql/plan/function/baseTemplate.go:553)

When the second operand is constant and the first vector has no input NULLs, this branch ignores rsAnyNull. A partial select list marks masked rows in rsNull, but the else loop still calls resultFn for them. The PR routes float subtraction and multiplication through this error-checking template, so a masked row such as MaxFloat64 * 2 or MaxFloat64 - (-MaxFloat64) raises ErrOutOfRange and aborts a CASE/IF query instead of producing NULL. The branch must honor the current rsNull mask (or include rsAnyNull in its guarded path).

P2 - Generic arithmetic still writes select-list masks past the batch (pkg/sql/plan/function/baseTemplate.go:609)

FunctionExpressionExecutor only grows its reusable SelectList; a later smaller batch retains stale tail entries. This loop still ranges over the entire retained SelectList instead of length, so a three-row mask reused for a two-row batch can add a NULL bit at row 2, outside the result vector. Nulls.Add expands the bitmap and Count includes that bit; for example, order/window operators can see nullCnt >= Length and skip sorting. The new bounded loops cover decimal, DIV, and MOD paths, but the generic plus/minus/multiply templates remain affected.

[XuPeng-SH]

There is still one blocking select-list correctness gap in the Decimal128 addition path.

P1 - Preserve mask-derived NULLs after Decimal128 addition (pkg/sql/plan/function/arithmetic.go:240)

decimalBatchArith now correctly adds unselected rows to the result NULL bitmap, but the Decimal128-result special case in plusFn unconditionally calls result.GetResultVector().GetNulls().Reset() whenever both input vectors contain no NULLs. That condition says nothing about select-list eligibility, so it erases the NULLs just established for masked CASE/IF rows.

I reproduced this at the current head bc96aa17c9: two non-NULL Decimal128 vectors, FunctionSelectList{AnyNull: true, SelectList: []bool{true, false}}, and a two-row result. plusFn returns nil, but result.GetNulls().Contains(1) is false; row 1 is exposed as a valid value although it was not selected.

Please remove the caller-side bitmap reset or make it preserve all result NULL sources, and add Decimal128 + partial-mask coverage (including vector/vector and const/vector or vector/const). The underlying invariant is that result validity is the union of input NULLs and execution-ineligible rows; it cannot be reconstructed from input NULL state alone.

The latest head does close the previously reported masked-overflow and stale-tail issues, and the current CI is green. This remaining case is absent from the matrix: the addition test uses Decimal64, while Decimal128 is covered only for /, DIV, and MOD.