Skip to content

HELP-1b — MATERIALIZED VIEW gets help documents, and the guard now runs parser to doc - #319

Merged
fupelaqu merged 1 commit into
mainfrom
feature/HELP-1b
Sep 9, 2026
Merged

HELP-1b — MATERIALIZED VIEW gets help documents, and the guard now runs parser to doc#319
fupelaqu merged 1 commit into
mainfrom
feature/HELP-1b

Conversation

@fupelaqu

@fupelaqu fupelaqu commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

HELP-1b — the guard now runs parser → doc, and MATERIALIZED VIEW gets its help

HelpCorpusSpec's eight pre-existing assertions all ran doc → parser. Nothing ran the other
way, so all nine MATERIALIZED VIEW productions could ship with zero help documents — the
largest hole in the corpus — while the suite stayed green.

What this adds

1. The first statement-level parser → doc assertion (HelpCorpusSpec, +152 lines, 3 new tests).

Statements are enumerated by walking the compiled AST package. Statement is sealed, so the
compiler guarantees every subtype is declared in one file and therefore compiled into one package
directory — the walk is complete, with no hand-written list of names (an allow-list is the artefact
that let MV drift in the first place, and it stops guarding the day statement 44 lands).

🔴 A scan of production RESULT TYPES was tried first and rejected by review. It cannot see
MultiSearch: searchStatement is rep1sep(single, union), so SELECT … UNION ALL SELECT …
yields a concrete, user-typeable statement produced by no def of its own. That version shipped
green with UNION ALL undocumented — the same class of hole the guard exists to close. Whenever a
production returns an abstract type, its leaves hide behind it.
The result-type scan survives as
(a) the failure clue (MultiSearch [(no production of its own - built inside a combinator)]) and
(b) an assertion that the package walk is a superset of it.

Every concrete SearchStatement maps to SELECT by a structural rule (SingleSearch,
MultiSearch, SelectStatement are its three leaves — a fourth resolves with no edit); the AST
result type also collapses all five createOrReplaceX/createX pairs for free. That leaves
FromlessSelect (#251's SELECT 1, deliberately not a SearchStatement) as the one named
exception, exactly as AD-3 predicted.

Four vacuity guards, because a reflection gate that stops reflecting goes GREEN, not red:

  • every method whose erased return type is a parser type must expose a ParameterizedType
    (if the generic Signature attribute stopped being emitted the enumeration would be empty);
  • the set of productions returning a sealed trait is asserted by exact set equality both
    ways — an inclusion test would let a new abstract-typed production join silently, and each one
    hides its leaves;
  • the package walk must be a superset of the result-type scan;
  • astStatementTypes must be non-empty, and not every type may claim to be a SELECT.

UNION ALL also gained a syntax line, a clause and an example in select.json, so the
MultiSearch → SELECT mapping is honest rather than merely green.

The assertion reads the source view only (sourceDocsUnder), per HELP-1a's AD-2b: here
documents are the resolution TARGET, so a stale target/ copy would REMOVE a failure.

2. Eight MATERIALIZED VIEW help documents (3 DDL + 5 DQL) with their three _index.json entries
(loadResourceDirectory reads only what the index names, so that update is load-bearing for the
shipped jar). Every one states the extension requirement, per #157.

3. documentation/sql/materialized_views.md reconciled with the grammar re-derived by execution.

Falsification of the new assertion (both halves run, both reverted)

  • Remove the 8 MV documents and their index entries ⇒ exactly one test red, naming all eight
    MV topics with the productions that produce them
    (CREATE MATERIALIZED VIEW <- createMaterializedView, createOrReplaceMaterializedView, …).
  • Remove truncate_table.json and its _index.json entry ⇒ the NEW assertion is red by name
    (TRUNCATE TABLE <- truncateTable) while the pre-existing index↔disk assertion stays green, so the
    two are distinguishable. (The seeAlso assertion also reddens — three documents point at
    TRUNCATE TABLE — which is correct behaviour, not interference.)
  • Remove select.json and its entry ⇒ red naming FromlessSelect, MultiSearch,
    SelectStatement, SingleSearch — the proof that the UNION ALL hole is closed.

⚠️ The guard does NOT cover command syntax templates — they were probed BY HAND

HelpCorpusSpec's syntax probe excludes command syntax arrays wholesale: they are one
pseudo-code block whose elements include "", ")" and -- ... lines. That exclusion is why
create_table.json's non-existent PARTITIONED BY survived HELP-1a. Every template this PR writes
or edits was therefore instantiated by hand and run through the real parser — transcript below.

T4 — hand instantiation of every command syntax template (the guard cannot see these)

HelpCorpusSpec's syntax probe excludes command syntax arrays WHOLESALE (they are one
pseudo-code block whose elements include "", ")" and -- ... lines). Every template below was
therefore instantiated BY HAND and run through the real Parser.

template instantiation verdict
create_materialized_view syntax[0-3] CREATE MATERIALIZED VIEW mv AS SELECT a FROM t ACCEPT
" (+IF NOT EXISTS) CREATE MATERIALIZED VIEW IF NOT EXISTS mv AS SELECT a FROM t ACCEPT
" (+REFRESH EVERY) CREATE MATERIALIZED VIEW mv REFRESH EVERY 30 SECONDS AS SELECT a FROM t ACCEPT
" (+WITH) CREATE MATERIALIZED VIEW mv WITH (delay = '1s') AS SELECT a FROM t ACCEPT
" (all optionals) CREATE MATERIALIZED VIEW IF NOT EXISTS mv REFRESH EVERY 60 SECONDS WITH (delay = '1s', user_latency = '1s') AS SELECT a FROM t ACCEPT
create_materialized_view syntax[6-9] CREATE OR REPLACE MATERIALIZED VIEW mv AS SELECT a FROM t ACCEPT
" (all optionals) CREATE OR REPLACE MATERIALIZED VIEW mv REFRESH EVERY 60 SECONDS WITH (delay = '1s', user_latency = '1s') AS SELECT a FROM t ACCEPT
drop_materialized_view syntax[0] DROP MATERIALIZED VIEW mv / ... IF EXISTS mv ACCEPT / ACCEPT
refresh_materialized_view syntax[0] REFRESH MATERIALIZED VIEW mv ACCEPT
" REFRESH MATERIALIZED VIEW IF EXISTS mv ACCEPT
" REFRESH MATERIALIZED VIEW mv WITH SCHEDULE NOW ACCEPT
" REFRESH MATERIALIZED VIEW IF EXISTS mv WITH SCHEDULE NOW ACCEPT
show_materialized_view syntax[0] SHOW MATERIALIZED VIEW mv ACCEPT
show_materialized_views syntax[0] SHOW MATERIALIZED VIEWS ACCEPT
show_materialized_view_status syntax[0] SHOW MATERIALIZED VIEW STATUS mv ACCEPT
show_create_materialized_view syntax[0] SHOW CREATE MATERIALIZED VIEW mv ACCEPT
describe_materialized_view syntax[0] DESCRIBE MATERIALIZED VIEW mv / DESC MATERIALIZED VIEW mv ACCEPT / ACCEPT
create_table syntax[0] (EDITED) CREATE TABLE IF NOT EXISTS t (id INT NOT NULL, PRIMARY KEY (id)) ACCEPT
create_table (new OR REPLACE line) CREATE OR REPLACE TABLE t (id INT NOT NULL) ACCEPT
create_pipeline syntax[0] (EDITED) CREATE PIPELINE IF NOT EXISTS p WITH PROCESSORS (SET (field = 'name', value = 'anonymous')) ACCEPT
create_pipeline (new OR REPLACE line) CREATE OR REPLACE PIPELINE p WITH PROCESSORS (SET (field = 'name', value = 'anonymous')) ACCEPT

Negative controls run in the same session (they must STAY rejected, and do):

statement verdict
CREATE OR REPLACE MATERIALIZED VIEW IF NOT EXISTS mv AS SELECT a FROM t REJECT — '(?i)AS\b' expected but 'N' found
CREATE MATERIALIZED VIEW mv WITH (delay = '1s') REFRESH EVERY 60 SECONDS AS SELECT a FROM t REJECT — clause order is fixed
SHOW MATERIALIZED VIEWS mv REJECT — end of input expected
SHOW MATERIALIZED VIEW IF EXISTS mv REJECT — no SHOW/DESCRIBE form takes IF EXISTS
DESCRIBE MATERIALIZED VIEW IF EXISTS mv REJECT
CREATE OR REPLACE TABLE IF NOT EXISTS t (id INT NOT NULL) REJECT — the form create_table.json used to publish
CREATE OR REPLACE PIPELINE IF NOT EXISTS p (...) REJECT — the form create_pipeline.json used to publish

Defects found while implementing, and FIXED here (nothing deferred)

  1. CreateMaterializedView.sql emitted no space before REFRESH EVERY, so
    SHOW CREATE MATERIALIZED VIEW handed back ... VIEW orders_mvREFRESH EVERY 8 SECONDS ...
    which re-parses as a view literally named orders_mvREFRESH and then fails. Every view with a
    refresh interval was affected, and MaterializedViewExtension persists that render. Fixed at the
    call site (Frequency.sql is untouched — TransformConfig renders it on its own line).
    Regression: six MV round-trip rows in ParserSpec, every one carrying a frequency, asserting
    Parser(stmt.sql) == Right(stmt) — not isRight, which stays green on the corrupt render. The
    pre-existing MV round-trip row in QuotedTableRoundTripSpec has no REFRESH EVERY, which is
    exactly why nothing saw this.
  2. create_table.json and create_pipeline.json published CREATE [OR REPLACE] X [IF NOT EXISTS] — measured REJECTED for both ('(?i)AS\b' expected but 'N' found). Same defect class
    as PARTITIONED BY, invisible to the guard for the same reason. Both rewritten into two forms,
    and both OR REPLACE clause descriptions now state the exclusion.
  3. DEFAULT CURRENT_TIMESTAMP in the markdown's CREATE TABLE example — not a default this
    engine accepts (defaultVal = value | ingest_id | ingest_timestamp). Rewritten to
    DEFAULT _ingest.timestamp. It occurs nowhere else in the repo.
  4. The SHOW CREATE "Returns:" block was elided with , ..., so the documented output of
    SHOW CREATE MATERIALIZED VIEW was not a statement anything accepts. Replaced with the real
    rendered form.
  5. "analytics"."orders_mv" and analytics.orders_mv create DIFFERENT indices, and the first
    draft of these documents presented them as interchangeable. Measured: a quoted qualifier is
    recorded but is not part of the name (view = orders_mv, per SQL parser: support backtick-qualified catalog names natively #85's "preserve, do not interpret"),
    while a bare dotted name is one legal index name (view = analytics.orders_mv). Both doc copies
    now say so; the misleading example is gone.
  6. REFRESH EVERY's time unit is case-SENSITIVE and needs whitespaceREFRESH EVERY 30 seconds
    and REFRESH EVERY 30SECONDS are both rejected in an otherwise case-insensitive dialect (the
    regex at Parser.scala:512-518 carries no (?i), unlike every keyword()). Nothing said so;
    now three places do. The real fix is a (?i) on that regex — SQL parser SHOW/DDL/DML keywords are case-sensitive #61's family, deliberately not
    widened here.
  7. The watcher note over-generalised: only a licence refusal degrades to a warning; a cluster
    with xpack.watcher.enabled: false returns 400 and hard-fails CREATE (extensions#49).
  8. Three guard-hygiene gaps found by review: nothing asserted category == directory (an
    unrecognised category silently becomes HelpCategory.Functions); a three-field stub satisfied
    every assertion, so command documents must now publish at least one example; and the enumeration
    matched only PackratParser, so a future Parser[T <: Statement] would have been invisible.

🔴 One defect this PR can only DOCUMENT, not fix — please rule on it

SHOW MATERIALIZED VIEWS (plural) is not implemented and always answers 400.
MaterializedViewExtension.canHandle accepts every MaterializedViewStatement, so the statement is
claimed, but execute has no branch for the plural form and falls into
case _ => "Unsupported statement for Materialized Views extension" (400). There is zero test
coverage anywhere. Both doc copies claimed it "returns a list of all materialized views".

The fix lives in softclient4es-extensions, which is not one of this story's referenced projects and
would need a metadata scan, a result shape and integration tests on five ES clients. So this PR does
the only honest thing available to it: it publishes the gap in the document's limitations and
notes and in both doc copies, instead of a claim that is false. No remote issue filed — that is
your call.

Verification

core/test 949/949 · sql/test 1035/1035 · HelpCorpusSpec 19/19 (14 pre-existing + 5 new) ·
sql/testOnly *DialectCensusSpec 16/16 · + core/Test/compile and + sql/compile green on 2.12
and 2.13 · headerCheck scalafmtCheck Test/scalafmtCheck scalafmtSbtCheck all green.

Release note

SHOW CREATE MATERIALIZED VIEW output changes for any view with a REFRESH EVERY clause: a space
now separates the view name from REFRESH. Downstream fixtures pinning that render need updating.

Companion docs PR: SOFTNETWORK-APP/softclient4es-web#56

Closes #318

…VIEW gets help

Closed Issue #318

Every assertion in HelpCorpusSpec ran doc -> parser, so all nine MATERIALIZED
VIEW productions shipped with no help document while the suite stayed green.

The new assertion enumerates statements by walking the compiled AST package:
`Statement` is sealed, so the compiler guarantees every subtype sits in one
package directory and the walk is complete with no allow-list. A scan of
production RESULT TYPES was tried first and rejected by review — it cannot see
`MultiSearch` (`UNION ALL`), which no production returns directly, and that is
the same class of hole the guard exists to close. The result-type scan survives
as the failure clue and as a superset assertion over the walk; the set of
productions returning a sealed trait is pinned by exact set equality, because
every one of them hides its leaves.

Eight MATERIALIZED VIEW documents (3 DDL + 5 DQL) with their three _index.json
entries — the index update is load-bearing for the shipped jar, not just for
the guard. `UNION ALL` is documented in select.json.

Fixed here rather than filed:
- CreateMaterializedView.sql emitted no space before REFRESH EVERY, so SHOW
  CREATE MATERIALIZED VIEW returned a statement no parser accepts. Six
  round-trip rows in ParserSpec, every one carrying a frequency, assert
  `Parser(stmt.sql) == Right(stmt)`.
- create_table.json and create_pipeline.json published
  `CREATE [OR REPLACE] X [IF NOT EXISTS]`, measured rejected.
- documentation/sql/materialized_views.md: the CREATE template, the elided
  SHOW CREATE output, `DEFAULT CURRENT_TIMESTAMP`, the qualifier semantics,
  the case-sensitive REFRESH EVERY unit, and the SHOW MATERIALIZED VIEWS gap.

The guard cannot see command `syntax` templates; all ten written or edited here
were instantiated by hand and run through the real parser.
@fupelaqu
fupelaqu marked this pull request as ready for review September 9, 2026 12:01
@fupelaqu
fupelaqu merged commit 29adc21 into main Sep 9, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MATERIALIZED VIEW has no help documents, and the corpus guard cannot see the gap

1 participant