Story 21.8 — conversion defects: CAST to BOOLEAN, temporal formats, and the DDL ingest path

[fupelaqu]

Story 21.8 of epic 21 (Core SQL front door). A conversion that cannot be performed is a bug, not a
no-op.

Closes #312

Lead-scoped to Parts A + B + E + G; C, D and F deferred. Four rulings were taken before any
code was written: scope, PD-1 (boolean semantics), OQ-3 (temporal formats), OQ-4 (unresolvable DDL
operand).

🔴 T0 refuted the story's own centrepiece

The spec shipped status: draft and said not to dev from it as written. Re-measuring every record
on current main moved two rows, and both reshaped the work.

Record 6 claimed four date functions emit uncompilable Painless — "the most serious" of the six —
and honestly flagged its blast-radius half as INFERRED. Discharging that inference first was the
highest-value hour of the story. The malformed e0def form is produced only by the context-free
painless(None). Every production surface renders these through painless(Some(ctx)), whose arm is
correct; the painless(None) sites in shipped code (bucket_selector, bucket_script, geo
distance, top_hits script fields) either carry aggregates or drop the field. No measured
production surface emits it ⇒ Part C deferred.

A different, LIVE defect was found in the same four functions and folded in. includeTimeZone
appended " XXX" to the CALLER'S pattern — measured on real ES 8.18.3:

statement	before	after
DATETIME_FORMAT(ts, 'yyyy')	"2025 Z" — silent wrong answer	"2025"
DATETIME_PARSE('2025-01-10 10:00:00', 'yyyy-MM-dd HH:mm:ss')	throws	2025-01-10T10:00Z

What changed

Part A — CAST(x AS BOOLEAN). Seven arms FROM boolean, none TO it, so every (_, Boolean) pair
fell to the identity fallback. C-style per PD-1: zero is false, Boolean.parseBoolean for strings.
Sources are ENUMERATED, not wildcarded, so (Boolean, Boolean) still reaches the identity arm.
canConvert is byte-identical and now pinned as deliberately disagreeing — it gates schema
evolution, a different contract.

Part B — the temporal format set. Painless has no expression-level try/catch (coerce
returns an expression embedded anywhere) and ofPattern's optional sections cannot express ISO's
variable-length fractional seconds, so a hand-written pattern would NARROW what parses. The set is
realised as a separator normalisation in front of a WIDER ISO formatter — one parse, strict
superset. TIMESTAMP moves to ISO_DATE_TIME.withZone(UTC), which fills a MISSING zone only: an
explicit +01:00 still wins (→ 13:30Z, measured).

Part G — the DDL ingest path. A computed column stored the UNCONVERTED operand into a correctly
mapped field. The spec designed two seams because ALTER … SET SCRIPT AS carries no column list —
true of the statement, false of the fix: Table.merge applies the ALTER to the table already
loaded from Elasticsearch, and CreateTable.schema builds its own, and both end at
Table.update(). One seam, both sites, no I/O, nothing added to the cost-probed parse path. So
AC-G2 is fixed rather than excluded.

Part E — no code. Records 3 (epoch millis) and 4 (DECIMAL scale) confirmed CORRECT and closed
with written reasoning, because "the doc was wrong so the engine must be wrong" is the obvious and
incorrect inference.

%f (second commit). Mapped to SSS under a comment saying "microseconds". The obvious fix
would have been a regression: S is FIXED WIDTH in both directions, so SSSSSS makes the docs true
by breaking the parse of .123, and [.SSSSSS][.SSS] formats as .123456.123. That is an "except"
inside the rule Part B is justified by. DateTimeFormatterBuilder is whitelisted (measured), so
%f emits a variable-width appendFraction: parses .123, .123456, .123456789 and no
fraction at all, and formats the value's real precision.

Verification

Every emission was EXECUTED on Elasticsearch before being pinned — a Painless claim is only
provable by Elasticsearch, and this project has paid for that rule twice. Part G is asserted on the
stored _source, never on the mapping and never on a read path: a correct mapping and a wrong
_source coexist silently.

• Unit: sql 945 · core 924 · macrosTests 21 · bridge 197 · es6bridge 197, both
  Scala legs
• Real ES: es6 rest 60/0/1 · es6 jest 60/0/1 · es7 61 · es8 61 · es9 61
• Falsification: 7 mutations, 7 predicted REDs, control green after restore

Release notes

• CAST(<n>|<string> AS BOOLEAN) now converts. ⚠️ CAST('1' AS BOOLEAN) is false — the numeric
  rule applies to a numeric operand, not to a numeric string. TRY_CAST to BOOLEAN can no longer
  yield NULL.
• CAST to DATE also accepts yyyy/MM/dd; to TIMESTAMP/DATETIME also the space-separated spelling,
  defaulting a missing zone to UTC; to TIME now ISO_LOCAL_TIME.
• DATETIME_FORMAT no longer appends a zone to the requested pattern and DATETIME_PARSE no longer
  demands one — values change for anyone relying on the old output.
• %f is variable width: it formats a value's real precision and parses any number of digits.
• A DDL computed column now CONVERTS on ingest. Existing indices keep their stored pipeline; the
  first ALTER after upgrading re-renders it.

Deferred, with local records (no new issues filed)

Parts C, D and F remain open in the spec. Two residuals were found and recorded locally:
date/time functions are unusable in a computed column (pre-existing, byte-identical after this
PR — that is AC-G3 holding — and on a published example), and it is routed to Part C, whose
family-wide guard should catch it. ⚠️ Not the same defect though: Part C is the context-free
assembly joining operand and call; the residual is the operand's runtime shape in processor context.
Part C's fix alone will not close it.

🤖 Generated with Claude Code

[fupelaqu]

CI caught a regression from part A — fixed in 70404137

The test job failed the same test on all five clients: GatewayApiIntegrationSpec › "should answer the FROM-less connection handshake against the live cluster", at the SELECT CASE WHEN 1 = 1 THEN 'a' ELSE 'b' END row.

What Elasticsearch received:

def param1 = (1 == 1 != null ? (def)((1 == 1 != 0)) : null); param1 ? "a" : "b"

search_phase_execution_exception: all shards failed; compile error
caused_by class_cast_exception: Cannot cast from [boolean] to [java.lang.Object].

Cause. Criteria fixes out to BOOLEAN but leaves baseType inherited from FunctionChain, which walks the identifier's chain — so 1 = 1 reported BIGINT and descr = 'x' reported the column's type. That has always been wrong and always been inert: with no (_, BOOLEAN) arm in coerce, every such pair fell to the identity fallback and the comparison was emitted untouched. Part A added the arms, and the lie became live.

It cannot be repaired inside coerce — once an arm matches, a genuine CAST(1 AS BOOLEAN) and this comparison are the same (BIGINT, BOOLEAN) pair. The source type has to be right before coerce is called.

Fix. One line in Criteria, beside the out override it belongs with:

override def baseType: SQLType = SQLTypes.Boolean

Emission is byte-for-byte restored to origin/main for every shape (literal comparison, numeric column, varchar column), so this pins a restoration, not a new shape.

The part worth flagging

The whole unit estate stayed green while every live client failed — 945 sql + 924 core + 2 × 197 bridge tests, and not one of them renders a CASE through a PainlessContext. BooleanCastSpec's helper used painless(None), and Case renders an entirely different way without a context: only the context branch hoists conditions into def paramN = … bindings, and only that branch coerces the condition. A context-free assertion could not have seen this no matter what it pinned. Same distinction the story's own T0 refutation turned on.

So the spec grows the production renderer it was missing, four byte pins, and the invariant itself (cond.baseType shouldBe BOOLEAN) so the next (_, BOOLEAN) arm cannot re-open the hole silently.

Falsified: reverting the one-line fix turns exactly those two tests red, and the mutated output reproduces the CI script byte-for-byte.

Re-verified on real Elasticsearch, GatewayApiSpec per client:

ES	suite
6.8 rest	72/72 (1 canceled)
6.8 jest	72/72 (1 canceled)
7.17	73/73
8.18.3	73/73
9.0	73/73

sql 947 · core 924 · bridge 197 · es6bridge 197 · scalafmtCheck · headerCheck · 2.12.20 sql/core Test/compile.

🤖 Generated with Claude Code