Story 21.7 — Parser.ident uniformity: quoted names for every DML/DDL statement kind

[fupelaqu]

Story 21.7 of epic 21 (Core SQL front door). Parser.ident — the third name surface — now has zero statement call sites: DQL, DML and DDL respect the same name rules.

Closes #310

What changed

Two productions replace ident at every one of its 78 call sites but two. Neither adds a lexer: identParts / identRef are built on 21.2's tableParts, identName on 21.1's qualifiedName.

production	built on	positions	AST
identRef (35 sites)	21.2 tableParts	table / view / pipeline / watcher / enrich-policy names	node carries parts: Seq[NamePart] = Nil, renders through the shared renderName
identName (41 sites)	21.1 qualifiedName	columns, INSERT column lists, UPDATE SET keys, PRIMARY KEY, conflict target, PARTITION BY, enrich FROM/ON/ENRICH, option and struct-entry keys, watcher input/action names	render decided by shape (renderColumnName)
ident (2 sites)	—	processorType, fileFormat — deliberate non-conversions	unchanged

Table.render now delegates to the shared sql.renderName, so there is ONE derivation of the rendering rather than two.

Task 0 — baseline, re-measured on origin/main 9f6726b2

The spec's baseline (ac54a079) is many merges stale, so it was re-measured. Every row the spec records as REJECT still rejected; both controls parsed. The split's premise is intact.

statement	verdict on 9f6726b2	rejection message
INSERT INTO `prod_eu`.dest SELECT a FROM src	REJECT	regex '[a-zA-Z_][a-zA-Z0-9_.]*' expected but '`' found
INSERT INTO "prod_eu".dest SELECT a FROM src	REJECT	same, " found
UPDATE `orders` SET a = 1 WHERE id = 1	REJECT	same, backtick found
CREATE TABLE "dest" ("c" INTEGER)	REJECT	same, " found
INSERT INTO tbl ("c") VALUES ('a')	REJECT	regex '(?i)(SELECT)\b' expected but '(' found
UPDATE tbl SET "a" = 1 WHERE id = 1	REJECT	end of input expected (differs from the spec's expectation; recorded as measured)
CREATE TABLE `#Tableau_sid_1_Connect_Chec` (`COL` INTEGER)	REJECT	ident regex
DROP TABLE IF EXISTS `#Tableau_sid_1_Connect_Chec`	REJECT	ident regex
CREATE TABLE dest (COL INTEGER)	PARSE	— (control)
CREATE LOCAL TEMPORARY TABLE dest (COL INTEGER)	REJECT	regex '(?i)OR\b' expected but 'L' found (control)
DELETE FROM "elastic".orders WHERE id = 1	PARSE	— (control; DELETE routes through FromParser.table)

Task 1.1 — classification of every ident site

80 \bident\b tokens in Parser.scala on 9f6726b2 = 78 code call sites + the definition + one prose mention. (The spec's "76" was measured at ac54a079; the count moved by 2 between those commits, as the spec warned it might.) 35 + 41 + 2 = 78, exactly.

Object reference → identRef (35), node carries parts (29 nodes)

family	productions
table (8)	createOrReplaceTable, createTable, showTable, showCreateTable, describeTable, dropTable, truncateTable, alterTable
materialized view (8)	createOrReplaceMaterializedView, createMaterializedView, dropMaterializedView, refreshMaterializedView, showMaterializedView, showMaterializedViewStatus, showCreateMaterializedView, describeMaterializedView
pipeline (7)	createOrReplacePipeline, createPipeline, dropPipeline, showPipeline, showCreatePipeline, describePipeline, alterPipeline
enrich policy (5)	createEnrichPolicy, createOrReplaceEnrichPolicy, executeEnrichPolicy, dropEnrichPolicy, showEnrichPolicy
watcher (4)	createOrReplaceWatcher, createWatcher (name only), showWatcherStatus, dropWatcher
DML target (3)	insert, copy, update

Column / key → identName (41): column, primaryKey, partitionBy, dropColumn, renameColumn (×2), the 15 alterColumn* column names plus DROP OPTION / DROP FIELD operands, DROP MAPPING / DROP SETTING / DROP ALIAS keys, dropProcessor column, the watcher WHEN field, chainInput and watcherAction names, enrich FROM / ON / ENRICH (×2 forms), conflictTarget, the INSERT column list, the UPDATE … SET left-hand sides, option keys and struct_entry keys.

Deliberate non-conversions (2), documented in source: processorType and fileFormat select an ENUM, not a name — IngestProcessor.sql renders an uppercase keyword, and fileFormat already accepts every quoted spelling through literal.

Design decisions

AD-1 — ident is retained as the LAST alternative of both new productions. tableParts reaches its name through bareFirstPart, which carries a reserved-word negative lookahead ident never had. A straight swap would newly reject ~130 bare names in every DDL/DML position — CREATE TABLE t (count INTEGER), DROP TABLE order, INSERT INTO t (min) … — a customer-visible regression with no migration path inside the release, which AC-5 forbids. Residual, stated: FROM count stays rejected while DROP TABLE count is accepted. That asymmetry is PRE-EXISTING and is not made worse; narrowing DDL onto DQL's reserved-word rule is a breaking change that needs scheduling, not a grammar clean-up.

AD-2 — one renderer. sql.renderName(parts, name) is the single implementation and Table.render delegates to it.

AD-3 — carry-or-pin, decided per node by measurement. The story's headline acceptance (#Tableau_sid_1_Connect_Chec) is exactly a name the bare surface cannot spell, so object references must carry parts. A name held inside a List[String] / ListMap[String, _] has nowhere to put a bit, so renderColumnName decides by shape — bare when the name matches bareNameRegex (which is now Parser.ident's own regex, its single owner), ANSI-quoted otherwise. Exact, not heuristic, precisely because of AD-1's fallback; and no reserved-word list is duplicated outside Parser.

AD-4 — parts is Nil for a single BARE part, normalised inside identRef. Such a reference IS its name, so recording it would make every parsed DDL node unequal to the programmatic one — TableDiff, IndicesApi and AlterTableRoundTripSpec all build AlterTable("dest", …) with no parts and compare against a re-parse.

AD-5 (OQ-1) — option and struct-entry keys ACCEPT quoting but render exactly as before. Measured on the unmodified tree: the double-quoted spelling of an OPTION key ALREADY parsed, through TypeParser.literal, and already gave the content as the key — so for option this is a backtick-only widening with a byte-identical AST. struct_entry had no literal alternative, so both spellings are new there.

AD-6 (post-review, lead ruling) — the story narrows NOTHING, on any surface. See below.

🔴 AD-6 — the reversal, and the defect it caught

The first implementation shipped one declared narrowing (DROP TABLE a. / a..b / SHOW TABLE a.) and, undeclared, the same narrowing on option and struct-entry keys:

statement	origin/main	first implementation
OPTIONS (a. = 1)	PARSE, key a.	REJECT
OPTIONS (a..b = 1)	PARSE, key a..b	REJECT

No test covered a malformed option key, so the branch's own 912-test suite was green over it. Only a parser built from origin/main and run against the same inputs could see it.

Cause, one shape in two places. identName = (qualifiedName ^^ (_._1)) | ident and identParts = tableParts | (ident …) both put the richer alternative first, and | commits to the first alternative that succeeds, not the one that consumes most. On a dangling dot the rich alternative succeeds on a PREFIX (a out of a.), leaving the rest unconsumed, so the enclosing sequence fails.

Ruling: fix both. A rule justified by "no regression in existing parsing" cannot hold for option keys and not for table names in the same story. Nobody means a. as an index name, but "nobody means it" is not a reason to stop accepting it inside a release — rejecting it is a breaking change and belongs to whoever schedules one.

Fix: <~ not(".") on both productions, so the rich alternative DECLINES exactly where it would commit to a prefix and ident takes the lexeme whole. It consumes nothing and cannot affect a well-formed name: every accepted shape ends at a delimiter, not at a dot. logs-2025.03, "sch".tbl and "elastic" . bi_events (21.2 AD-8's whitespace-tolerant qualifier dot) are untouched, and the FROM surface never reaches these productions. The three narrowing pins were retargeted, never deleted, and the option keys gained the coverage whose absence hid the defect.

Differential probe, 28 option/struct shapes against origin/main: 25 byte-identical, 3 widened, 0 narrowed.

🔴 The durable lesson: a production's alternation ORDER is a contract about what it DECLINES, not only about what it accepts. Swapping a first alternative for a richer one silently narrows every input the richer one matches a PREFIX of — and a branch-only suite cannot see it, because the inputs it breaks are exactly the ones nobody wrote a test for.

Verification

Regression measured, not claimed. 2,049 statements harvested from the repo's own test sources, pushed through parsers compiled from BOTH trees (the branch, and origin/main restored into the same committed worktree — a genuine single-variable control):

• 32 newly PARSE;
• 0 newly REJECT (the 3 malformed-dotted-name rows the first implementation rejected are restored by AD-6);
• render CHANGED among both-parse statements: 0;
• the 12 render-re-parse rejections and 44 AST-differs are byte-for-byte the same set on both trees — the pre-existing NOT <op> / IN fix(sql): make ALTER COLUMN ... SET|ADD FIELD parse and actually apply #218 family, 21.5's array DEFAULT render, ALTER PIPELINE's processor-field AST and ORDER BY resolution. Zero introduced.

AC-6, measured on the real corpus. Replaying the 24 DDL rows of epic-19-bi-corpus.csv gives exactly the split 21.6's attribution predicts: the 21 capability_open rows parse, and the 3 rejected_pending_policy rows (CREATE LOCAL TEMPORARY TABLE) still reject with regex '(?i)OR\b' expected but 'L' found. Recorded here for 21.6 to consume; these rows are never scored as wins and no headline number is published from them — 21.6 owns the number. Whether to honour a temp-table capability probe remains a separate product decision about CREATE TABLE semantics, tracked outside this epic.

OQ-2 — what a real Elasticsearch actually did, not what was expected. A disposable Elasticsearch 8.18.3 container, both probes through GatewayApi.run:

statement	observed
CREATE TABLE `#Tableau_sid_1_Connect_Chec` (`COL` INTEGER)	ElasticFailure, status 400, operation=Some("schema"), Invalid index: Index name must be lowercase
DROP TABLE IF EXISTS `#Tableau_sid_1_Connect_Chec`	ElasticFailure, status 400, operation=Some("drop"), same message
CREATE TABLE `#tableau_probe_lower` (`COL` INTEGER) (isolating the variable)	status 400, Invalid index: Index name contains invalid characters: /, *, ?, ", <, >, |, space, comma, #
CREATE TABLE plain_probe_control (COL INTEGER) / DROP TABLE IF EXISTS …	SUCCESS — the cluster and the path are live

The expectation ("# is not a legal ES index-name character") was right about the outcome and wrong about the mechanism: the refusal is client-side, in SoftClient4ES's own index-name validator, before the request reaches Elasticsearch, and the first reason reported is the uppercase rule. Not wired into CI.

Documentation. Every SQL example added to documentation/sql/dql_statements.md, and the two documentation/sql/joins.md:250-265 DDL examples that only become true here, were parse-probed against the built parser and round-tripped.

Suites. sql 914 and core 924 on both Scala legs, bridge 197, es6bridge 197, macrosTests 21, DialectCensusSpec 16 unmoved, scalafmtCheckAll green. ++ 2.12.20 sql/Test/compile green. Parse cost main 5,935 ms → branch 5,799 ms over 2,049 statements (−2.3 %, overlapping ranges), well inside the #269 gate.

Not run, deliberately: sql/Test/headerCheck (RED on main; new sql test files take no header) and the Docker/ES suites (nothing here reaches a client — OQ-2 was verified once by hand).

Behaviour changes

#	shape	kind
U1	INSERT INTO `prod_eu`.dest / INSERT INTO "prod_eu".dest — target dest, qualifier captured in parts, never interpreted	widening
U2	UPDATE `orders` SET …, CREATE TABLE "dest" ("c" INTEGER), and the same for DROP / TRUNCATE / ALTER / COPY INTO / SHOW / DESCRIBE / MV / pipeline / watcher / enrich-policy names	widening
U3	quoted COLUMN names — INSERT INTO tbl ("c") VALUES ('a'), UPDATE tbl SET "a" = 1, ALTER TABLE … ALTER COLUMN "c" …	widening
U4	not in the spec's table, found by measurement — a hyphenated or bracketed name is now spellable in DDL/DML: CREATE TABLE logs-2025.03, DROP TABLE my-index, INSERT INTO logs-2025.03. SELECT and DELETE already accepted them; this closes the last surface that did not	widening
U5	a backticked option key, and BOTH quoted spellings of a struct-entry key. The double-quoted OPTION key already parsed via TypeParser.literal — byte-identical AST, measured. A hyphenated key (Content-Type) was never spellable in SQL at all	widening

No narrowings. Nothing that parses on origin/main stops parsing.

Pre-existing, unchanged, and NOT introduced here (recorded so it is not mistaken for a regression): OPTIONS ('my key' = 1) renders OPTIONS (my key = 1) on origin/main and on this branch — a key the bare surface cannot spell renders bare and does not re-parse. That is the #218 render family.

Release note

INSERT / UPDATE / CREATE / DROP / TRUNCATE / ALTER / COPY INTO and every SHOW/DESCRIBE now accept quoted table AND column names (both quote styles) with the same qualifier semantics as SELECT, and a hyphenated name (logs-2025.03, my-index) is spellable there too — a pure widening; bare spellings parse to a byte-identical AST and render. Backticked option keys and quoted struct-entry keys are accepted; keys still render exactly as before. 29 DDL/DML AST nodes gained a defaulted parts field: binary-incompatible (case-class arity — the same rebuild 21.1/21.2 already owe on this train) and source-incompatible for case <Node>(…) destructuring.

Standing product questions (neither blocking, neither introduced here)

1. Should DDL/DML adopt DQL's reserved-word rule? FROM count is rejected while DROP TABLE count is accepted — pre-existing, not made worse. Closing it breaks ~130 bare names with no migration path, so it is a scheduled deprecation.
2. Should an option/struct-entry key the bare surface cannot spell be rendered quoted? Pre-existing and identical on both trees; closing it moves the rendered text of every watcher, pipeline and OPTIONS clause with such a key.

🤖 Generated with Claude Code