HELP-1b — MATERIALIZED VIEW gets help documents, and the guard now runs parser to doc - #319
Merged
Conversation
…VIEW gets help Closed Issue #318 Every assertion in HelpCorpusSpec ran doc -> parser, so all nine MATERIALIZED VIEW productions shipped with no help document while the suite stayed green. The new assertion enumerates statements by walking the compiled AST package: `Statement` is sealed, so the compiler guarantees every subtype sits in one package directory and the walk is complete with no allow-list. A scan of production RESULT TYPES was tried first and rejected by review — it cannot see `MultiSearch` (`UNION ALL`), which no production returns directly, and that is the same class of hole the guard exists to close. The result-type scan survives as the failure clue and as a superset assertion over the walk; the set of productions returning a sealed trait is pinned by exact set equality, because every one of them hides its leaves. Eight MATERIALIZED VIEW documents (3 DDL + 5 DQL) with their three _index.json entries — the index update is load-bearing for the shipped jar, not just for the guard. `UNION ALL` is documented in select.json. Fixed here rather than filed: - CreateMaterializedView.sql emitted no space before REFRESH EVERY, so SHOW CREATE MATERIALIZED VIEW returned a statement no parser accepts. Six round-trip rows in ParserSpec, every one carrying a frequency, assert `Parser(stmt.sql) == Right(stmt)`. - create_table.json and create_pipeline.json published `CREATE [OR REPLACE] X [IF NOT EXISTS]`, measured rejected. - documentation/sql/materialized_views.md: the CREATE template, the elided SHOW CREATE output, `DEFAULT CURRENT_TIMESTAMP`, the qualifier semantics, the case-sensitive REFRESH EVERY unit, and the SHOW MATERIALIZED VIEWS gap. The guard cannot see command `syntax` templates; all ten written or edited here were instantiated by hand and run through the real parser.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
HELP-1b — the guard now runs parser → doc, and MATERIALIZED VIEW gets its help
HelpCorpusSpec's eight pre-existing assertions all ran doc → parser. Nothing ran the otherway, so all nine
MATERIALIZED VIEWproductions could ship with zero help documents — thelargest hole in the corpus — while the suite stayed green.
What this adds
1. The first statement-level parser → doc assertion (
HelpCorpusSpec, +152 lines, 3 new tests).Statements are enumerated by walking the compiled AST package.
Statementissealed, so thecompiler guarantees every subtype is declared in one file and therefore compiled into one package
directory — the walk is complete, with no hand-written list of names (an allow-list is the artefact
that let MV drift in the first place, and it stops guarding the day statement 44 lands).
🔴 A scan of production RESULT TYPES was tried first and rejected by review. It cannot see
MultiSearch:searchStatementisrep1sep(single, union), soSELECT … UNION ALL SELECT …yields a concrete, user-typeable statement produced by no
defof its own. That version shippedgreen with
UNION ALLundocumented — the same class of hole the guard exists to close. Whenever aproduction returns an abstract type, its leaves hide behind it. The result-type scan survives as
(a) the failure clue (
MultiSearch [(no production of its own - built inside a combinator)]) and(b) an assertion that the package walk is a superset of it.
Every concrete
SearchStatementmaps toSELECTby a structural rule (SingleSearch,MultiSearch,SelectStatementare its three leaves — a fourth resolves with no edit); the ASTresult type also collapses all five
createOrReplaceX/createXpairs for free. That leavesFromlessSelect(#251'sSELECT 1, deliberately not aSearchStatement) as the one namedexception, exactly as AD-3 predicted.
Four vacuity guards, because a reflection gate that stops reflecting goes GREEN, not red:
ParameterizedType(if the generic
Signatureattribute stopped being emitted the enumeration would be empty);ways — an inclusion test would let a new abstract-typed production join silently, and each one
hides its leaves;
astStatementTypesmust be non-empty, and not every type may claim to be aSELECT.UNION ALLalso gained asyntaxline, a clause and an example inselect.json, so theMultiSearch → SELECTmapping is honest rather than merely green.The assertion reads the source view only (
sourceDocsUnder), per HELP-1a's AD-2b: heredocuments are the resolution TARGET, so a stale
target/copy would REMOVE a failure.2. Eight MATERIALIZED VIEW help documents (3 DDL + 5 DQL) with their three
_index.jsonentries(
loadResourceDirectoryreads only what the index names, so that update is load-bearing for theshipped jar). Every one states the extension requirement, per #157.
3.
documentation/sql/materialized_views.mdreconciled with the grammar re-derived by execution.Falsification of the new assertion (both halves run, both reverted)
MV topics with the productions that produce them
(
CREATE MATERIALIZED VIEW <- createMaterializedView, createOrReplaceMaterializedView, …).truncate_table.jsonand its_index.jsonentry ⇒ the NEW assertion is red by name(
TRUNCATE TABLE <- truncateTable) while the pre-existing index↔disk assertion stays green, so thetwo are distinguishable. (The
seeAlsoassertion also reddens — three documents point atTRUNCATE TABLE— which is correct behaviour, not interference.)select.jsonand its entry ⇒ red namingFromlessSelect,MultiSearch,SelectStatement,SingleSearch— the proof that theUNION ALLhole is closed.syntaxtemplates — they were probed BY HANDHelpCorpusSpec's syntax probe excludes commandsyntaxarrays wholesale: they are onepseudo-code block whose elements include
"",")"and-- ...lines. That exclusion is whycreate_table.json's non-existentPARTITIONED BYsurvived HELP-1a. Every template this PR writesor edits was therefore instantiated by hand and run through the real parser — transcript below.
T4 — hand instantiation of every command
syntaxtemplate (the guard cannot see these)HelpCorpusSpec's syntax probe excludes commandsyntaxarrays WHOLESALE (they are onepseudo-code block whose elements include
"",")"and-- ...lines). Every template below wastherefore instantiated BY HAND and run through the real
Parser.CREATE MATERIALIZED VIEW mv AS SELECT a FROM tCREATE MATERIALIZED VIEW IF NOT EXISTS mv AS SELECT a FROM tCREATE MATERIALIZED VIEW mv REFRESH EVERY 30 SECONDS AS SELECT a FROM tCREATE MATERIALIZED VIEW mv WITH (delay = '1s') AS SELECT a FROM tCREATE MATERIALIZED VIEW IF NOT EXISTS mv REFRESH EVERY 60 SECONDS WITH (delay = '1s', user_latency = '1s') AS SELECT a FROM tCREATE OR REPLACE MATERIALIZED VIEW mv AS SELECT a FROM tCREATE OR REPLACE MATERIALIZED VIEW mv REFRESH EVERY 60 SECONDS WITH (delay = '1s', user_latency = '1s') AS SELECT a FROM tDROP MATERIALIZED VIEW mv/... IF EXISTS mvREFRESH MATERIALIZED VIEW mvREFRESH MATERIALIZED VIEW IF EXISTS mvREFRESH MATERIALIZED VIEW mv WITH SCHEDULE NOWREFRESH MATERIALIZED VIEW IF EXISTS mv WITH SCHEDULE NOWSHOW MATERIALIZED VIEW mvSHOW MATERIALIZED VIEWSSHOW MATERIALIZED VIEW STATUS mvSHOW CREATE MATERIALIZED VIEW mvDESCRIBE MATERIALIZED VIEW mv/DESC MATERIALIZED VIEW mvCREATE TABLE IF NOT EXISTS t (id INT NOT NULL, PRIMARY KEY (id))CREATE OR REPLACE TABLE t (id INT NOT NULL)CREATE PIPELINE IF NOT EXISTS p WITH PROCESSORS (SET (field = 'name', value = 'anonymous'))CREATE OR REPLACE PIPELINE p WITH PROCESSORS (SET (field = 'name', value = 'anonymous'))Negative controls run in the same session (they must STAY rejected, and do):
CREATE OR REPLACE MATERIALIZED VIEW IF NOT EXISTS mv AS SELECT a FROM t'(?i)AS\b' expected but 'N' foundCREATE MATERIALIZED VIEW mv WITH (delay = '1s') REFRESH EVERY 60 SECONDS AS SELECT a FROM tSHOW MATERIALIZED VIEWS mvend of input expectedSHOW MATERIALIZED VIEW IF EXISTS mvIF EXISTSDESCRIBE MATERIALIZED VIEW IF EXISTS mvCREATE OR REPLACE TABLE IF NOT EXISTS t (id INT NOT NULL)create_table.jsonused to publishCREATE OR REPLACE PIPELINE IF NOT EXISTS p (...)create_pipeline.jsonused to publishDefects found while implementing, and FIXED here (nothing deferred)
CreateMaterializedView.sqlemitted no space beforeREFRESH EVERY, soSHOW CREATE MATERIALIZED VIEWhanded back... VIEW orders_mvREFRESH EVERY 8 SECONDS ...—which re-parses as a view literally named
orders_mvREFRESHand then fails. Every view with arefresh interval was affected, and
MaterializedViewExtensionpersists that render. Fixed at thecall site (
Frequency.sqlis untouched —TransformConfigrenders it on its own line).Regression: six MV round-trip rows in
ParserSpec, every one carrying a frequency, assertingParser(stmt.sql) == Right(stmt)— notisRight, which stays green on the corrupt render. Thepre-existing MV round-trip row in
QuotedTableRoundTripSpechas noREFRESH EVERY, which isexactly why nothing saw this.
create_table.jsonandcreate_pipeline.jsonpublishedCREATE [OR REPLACE] X [IF NOT EXISTS]— measured REJECTED for both ('(?i)AS\b' expected but 'N' found). Same defect classas
PARTITIONED BY, invisible to the guard for the same reason. Both rewritten into two forms,and both
OR REPLACEclause descriptions now state the exclusion.DEFAULT CURRENT_TIMESTAMPin the markdown'sCREATE TABLEexample — not a default thisengine accepts (
defaultVal = value | ingest_id | ingest_timestamp). Rewritten toDEFAULT _ingest.timestamp. It occurs nowhere else in the repo.SHOW CREATE"Returns:" block was elided with, ..., so the documented output ofSHOW CREATE MATERIALIZED VIEWwas not a statement anything accepts. Replaced with the realrendered form.
"analytics"."orders_mv"andanalytics.orders_mvcreate DIFFERENT indices, and the firstdraft of these documents presented them as interchangeable. Measured: a quoted qualifier is
recorded but is not part of the name (view =
orders_mv, per SQL parser: support backtick-qualified catalog names natively #85's "preserve, do not interpret"),while a bare dotted name is one legal index name (view =
analytics.orders_mv). Both doc copiesnow say so; the misleading example is gone.
REFRESH EVERY's time unit is case-SENSITIVE and needs whitespace —REFRESH EVERY 30 secondsand
REFRESH EVERY 30SECONDSare both rejected in an otherwise case-insensitive dialect (theregex at
Parser.scala:512-518carries no(?i), unlike everykeyword()). Nothing said so;now three places do. The real fix is a
(?i)on that regex — SQL parser SHOW/DDL/DML keywords are case-sensitive #61's family, deliberately notwidened here.
with
xpack.watcher.enabled: falsereturns 400 and hard-fails CREATE (extensions#49).category== directory (anunrecognised category silently becomes
HelpCategory.Functions); a three-field stub satisfiedevery assertion, so command documents must now publish at least one example; and the enumeration
matched only
PackratParser, so a futureParser[T <: Statement]would have been invisible.🔴 One defect this PR can only DOCUMENT, not fix — please rule on it
SHOW MATERIALIZED VIEWS(plural) is not implemented and always answers 400.MaterializedViewExtension.canHandleaccepts everyMaterializedViewStatement, so the statement isclaimed, but
executehas no branch for the plural form and falls intocase _ => "Unsupported statement for Materialized Views extension"(400). There is zero testcoverage anywhere. Both doc copies claimed it "returns a list of all materialized views".
The fix lives in
softclient4es-extensions, which is not one of this story's referenced projects andwould need a metadata scan, a result shape and integration tests on five ES clients. So this PR does
the only honest thing available to it: it publishes the gap in the document's
limitationsandnotesand in both doc copies, instead of a claim that is false. No remote issue filed — that isyour call.
Verification
core/test949/949 ·sql/test1035/1035 ·HelpCorpusSpec19/19 (14 pre-existing + 5 new) ·sql/testOnly *DialectCensusSpec16/16 ·+ core/Test/compileand+ sql/compilegreen on 2.12and 2.13 ·
headerCheck scalafmtCheck Test/scalafmtCheck scalafmtSbtCheckall green.Release note
SHOW CREATE MATERIALIZED VIEWoutput changes for any view with aREFRESH EVERYclause: a spacenow separates the view name from
REFRESH. Downstream fixtures pinning that render need updating.Companion docs PR: SOFTNETWORK-APP/softclient4es-web#56
Closes #318