fix(21.8 Part F.1): an ALTER must not rewrite ingest processors it was not asked to touch

[fupelaqu]

Closes #317

Story 21.8 Part F.1, the last open part of the story. Base origin/main 011bc912.

What was wrong

Every ALTER re-created ingest processors nobody had touched. The local record named one cause; building the gate it asked for — "no test asserts the pipeline diff is empty for an unchanged processor" — found three. Each is reachable from a different processor kind, so no single statement could ever exhibit more than one, which is why one example had only ever shown one.

#	cause	reachable on
1	an anonymous script processor has no column, so it cannot match its declared counterpart	ES 6.8 only
2	a stored non-textual value reads back as a STRING, so a numeric/boolean DEFAULT reports changed forever	every version
3	the FINAL pipeline is diffed against the DEFAULT one	any table with a final pipeline

(2) should change how this class is hunted. It never depended on description, so it was never confined to 6.8 — and the warning it produced was ProcessorPropertyChanged(value, 0.0, 0.0), a diff that prints as no diff. It survived every version run because it was unreadable, not because it was rare.

The fix

• ScriptTarget — one object owning the ingest assignment format. assign writes ctx.<column> = …, of reads it back, reference is shared with Column.update, which spelled ctx.<name> a third way.
• IngestPipeline.diff's KEY resolves a script processor's target through it, on both sides. The identity was in the script all along; description was never its only carrier.
• IngestProcessor.readValue — the typed read IndexField already uses for null_value, with Value.unwrap as its inverse on the write side.
• Table.declaredPipeline(pipelineType) — exhaustive; GatewayApi.loadTablePipelineDiff calls it instead of always reading defaultPipeline.

Container-valued SET processors are fixed with it: a list or object was lost to the empty string on read-back and now round-trips.

Why the recovery lives in the diff key and not in column

The first attempt put it in GenericProcessor.column, which is read by Table.defaultPipeline's filterNot, IngestPipeline.merge, ProcessorRemoved.stmt and describe. Measured: a hand-added ADD PROCESSOR SCRIPT(ctx.age = 1) then collided with a declared age SCRIPT AS (…) column, was filtered out of the desired pipeline, and the next ALTER emitted a DROP for it — data loss, to fix a cosmetic diff.

Confined to the key it reaches the only consumer that needs it, and because both sides derive the identity the same way from the same string, a source of misreads yields the same key on both sides and so cannot manufacture a difference.

Why Value.unwrap is needed alongside the typed read

Values.value and ObjectValue.value hold wrapped elements, and properties feeds them to mapToJsonNode. Measured: a list read back and written out again became "value":[{"value":"a","regex":{…},"expr":{…}}] — and ALTER PIPELINE rewrites every processor of the merged pipeline, so an untouched processor would be written back to the cluster in that shape. unwrap recurses, because one level left a nested array still serialising as beans.

A list whose head is Null has no arm in the sealed Values hierarchy and made Value.apply throw inside an uncaught path, so it keeps the old text read rather than killing the ALTER.

Proof

GatewayApiIntegrationSpec gained "report NO changes when an ALTER leaves every processor untouched": re-issue an ALTER that changes nothing and require DdlResult(false), which the engine emits only for an empty diff, with DdlResult(true) on the first call as a positive control. Reverting the sql fixes reds it on real Elasticsearch 6.8.23 with the record's own line, two causes at once:

🔄 Default ingesting pipeline for index users_alter_churn has changes: List(
  ProcessorAdded(seniority INT SCRIPT AS (DATE_DIFF(join_date, CURRENT_DATE, DAY))),
  ProcessorRemoved(SCRIPT(lang = "painless", source = "def param1 = ctx.join_date; …")),
  ProcessorChanged(reputation SET DEFAULT '0.0', reputation SET DEFAULT 0.0,
                   ProcessorDiff(None, List(ProcessorPropertyChanged(value, 0.0, 0.0)))))

assertDdl could never have caught this: a churning ALTER succeeds.

Verification

sql 1029 · core 944 · bridge 197 · es6bridge 197 · macrosTests 21. Root sbt test: es6 rest 400, es6 jest 385, es7 403, es8 419, es9 413 — 0 failed, the new gate on all five clients, and zero ProcessorRemoved(SCRIPT( and zero ProcessorChanged( across the whole run. Scala 2.12 leg compiles; scalafmt and headerCheck clean.

Nine mutations, nine predicted REDs, control green, sources byte-identical after restore.

Extensions verified against a local publish: materialized views green on ES 6/7/8/9 (ExtensionsIntegrationSpec and MaterializedViewBasicLicenseSpec per major, plus generator/metadata/quota/watcher/if-exists). Full extensions suite exits 0.

⚠️ sbt exits 1 locally because four ES 6.8 Jest suites abort before running — the embedded (non-Docker) Elasticsearch cannot launch its own JVM on an arm64 host (Unrecognized VM option 'UseAVX=2'). Single-variable control, es6jest/test on each tree: pristine origin/main 384 passed / 3 aborted, this branch 385 passed / 3 aborted. Same three, same cause, both trees.

Release notes

• An ALTER no longer rewrites ingest processors it was not asked to touch. A table with a computed column (ES 6.8) or a non-textual DEFAULT (every version) had its default pipeline re-created on every ALTER, with the standing "Default Pipeline may be used by multiple indexes" warning each time.
• A table with a FINAL pipeline no longer has its final processors compared against its default ones.
• A SET processor's stored value keeps its JSON type when read back, containers included, so SHOW TABLE and DESCRIBE report a list, object, number or boolean as itself rather than as a string or "".
• No binary-incompatible change: additions only.

Recorded, not fixed

• A fourth divergence: IngestPipeline(name, json, …) stamps the pipeline's type onto its processors, the case-class constructor does not, so a CUSTOM pipeline's two sides disagree. Unreached by shipped code (PipelineApi merges custom pipelines). Pinned on the cause, not on the size of the diff.
• A stored final SCRIPT processor still diffs as removed — a table never declares one, because the Index -> Table load attaches a final pipeline's ScriptProcessor to its column. Needs the load path, not the diff.
• Two script processors targeting the same column collide in the diff key (toMap drops one). Not new on ES 7+, where ScriptDescRegex has always keyed them that way; this aligns 6.8 with the rest.
• ScriptProcessor.fromScript splits on ; with no string-literal awareness, and its multi-statement branch never strips a trailing return. Both pre-existing.
• Record item 2 (balance alterPipeline, audit alterTable) not done — reclassified by the spec as a separate concern; both start.? ~ … ~ end.? sites remain.

Review

Three independent fresh-context layers. They found that the first attempt would have dropped a hand-added script processor, and that it had silently disarmed story 21.5's gate — AlterPipelineRenderSpec's ctx.a = 1 fixture resolved to a, so the content-addressed fallback that story shipped was never exercised and its falsification would now score green. Both fixed; the fixtures now use a form the recovery declines, and the file asserts it still reaches the fallback.

🤖 Generated with Claude Code