Skip to content

fix(copilot-on-rails): require quoted mermaid labels in debug plan diagrams - #1813

Merged
Nathan (nturinski) merged 8 commits into
feat/CoRfrom
nturinski-mermaid-label-quoting-490
Sep 14, 2026
Merged

Nathan (nturinski) merged 8 commits into
feat/CoRfrom
nturinski-mermaid-label-quoting-490

Conversation

@nturinski

Copy link
Copy Markdown
Member

Base branch: feat/CoRnot main. The Copilot on Rails code does not exist on main.

Problem

The azure-debug-plan agent generated an ## Architecture Diagram in .azure/vscode-debug-plan.md containing this edge:

API -->|@azure/storage-blob| AZ

Mermaid v11 reserves @ for edge-ID syntax (e1@-->) and node-metadata syntax (id@{ shape: ... }), so a @ at the start of an unquoted label token is lexed as LINK_ID and the whole diagram fails to parse:

Parse error on line 15:
...|pg| PG    API -->|@azure/storage-blob
---------------------^
Expecting 'AMP', 'COLON', 'PIPE', 'TESTSTR', 'DOWN', 'DEFAULT', 'NUM', 'COMMA',
'NODE_STRING', 'BRKT', 'MINUS', 'MULT', 'UNICODE_TEXT', got 'LINK_ID'

LocalPlanView's MermaidBlock catches the throw from mermaid.render() and falls back to a raw code block labeled mermaid (error) — so the user sees diagram source instead of the diagram.

Verified behavior (headless, jsdom, real mermaid builds):

Diagram Result
A-->|@azure/storage-blob| B ❌ fails
A[@azure/storage-blob]-->B ❌ fails — node labels affected too
A-->|uses @azure/storage-blob| B ✅ passes — @ mid-label is fine
A-->|"@azure/storage-blob"| B ✅ passes — quoting fixes it

Only a leading @ breaks. Scoped npm package names (@azure/storage-blob, @azure/identity, @prisma/client) are exactly what the agent puts in Azure-dependency edge labels, so this is very reachable in practice.

Not a regression

Reproduced identically on 11.14.0 (the version originally added in #1436), 11.15.0 (the dependabot bump in #1439), and 11.17.0 (what resolves today for "mermaid": "^11.15.0"). The LINK_ID grammar predates the CoR mermaid dependency entirely.

package.json and package-lock.json are deliberately untouched — bumping or pinning mermaid would not fix this.

Other characters in the failing diagram (<br/>, the · middle dot, the [( )] cylinder shape, :5173 port suffixes, blank lines between statements) all parse fine and were ruled out.

Fix

Harden the agent-facing authoring guidance so generated diagrams always quote labels.

  • resources/agents/azure-debug-plan/references/plan-template.md — new ## ⚠️ Mermaid Label Quoting — ALWAYS Quote Every Label section alongside the existing ## Markdown Table Integrity rule (outside the ```markdown template fence, so it reads as an agent rule rather than template body). It carries a ✅/❌ example pair and a concise explanation of the LINK_ID cause. Inline %% hints were also added inside the template's Architecture Diagram block so the rule is visible at the exact point of authoring, not just in a section the model might skim past.
  • .github/instructions/copilot-on-rails-docs.instructions.md — the same rule attached to the existing "keep the Mermaid pipeline diagram accurate" bullet, since that file also instructs an agent to maintain a mermaid diagram (the docs/copilot-create-project.md pipeline flowchart).

The rule is unconditional — quote every node label and every edge label, always — rather than an @-specific special case. Quoting is always valid in mermaid and additionally protects (, ), :, /, #, and ·. Asking the model to detect which labels are "risky" is far less robust than a rule it applies every time.

Intentionally not changed: resources/agents/azure-project-plan.agent.md and resources/agents/azure-project-plan/plan.md mention mermaid only to prohibit it in the parsed project plan, so a quoting rule there would be wrong.

Lockfile note

evals/agent-assets.lock.json is refreshed via node evals/check-agent-drift.ts --update, because plan-template.md is a hash-tracked agent asset and editing it fails the drift gate. Only that file's hash, the rollup agentAssetsHash, and updatedAt changed — no scope or tracked-file-set change.

Validation

The full credential-free contracts CI job was run locally against fc079fd; all 12 gates pass:

drift · typecheck · imports:self-test · certify (152/152 grader certification cases) · stacks:check · gates · phases:check · seed:contract · lint · clean-machine:check

lint reports 3 pre-existing scoring-defaults-applied warnings that are unrelated to this change.

Parse behavior was confirmed directly against real mermaid builds in a throwaway sandbox outside the repo: the unquoted form fails, the quoted replacements parse. The four existing fixture plans under evals/grader-certification/** and test/testProjects/copilotOnRails/** were each checked and already parse cleanly today (none has a leading @), so they were left alone rather than churned.

The behavioral MSBench evals are workflow_dispatch-only (they need Entra auth, not GITHUB_TOKEN) and are being dispatched separately against this PR.

…agrams

The azure-debug-plan agent emitted an Architecture Diagram edge label of
`API -->|@azure/storage-blob| AZ`. Mermaid v11 reserves `@` for edge-ID
(`e1@-->`) and node-metadata (`id@{ shape: ... }`) syntax, so a leading
`@` in an unquoted label lexes as LINK_ID and the whole diagram fails to
parse -- LocalPlanView's MermaidBlock then falls back to a raw
`mermaid (error)` code block instead of the diagram.

Verified headlessly against mermaid 11.15.0: the unquoted form fails, the
quoted form parses. Not a version regression (same on 11.14/11.15/11.17),
so the dependency is left alone.

- resources/agents/azure-debug-plan/references/plan-template.md: add an
  unconditional `Mermaid Label Quoting` rule with correct/incorrect
  examples and the reason, plus inline %% hints in the template's
  Architecture Diagram block.
- .github/instructions/copilot-on-rails-docs.instructions.md: same quoting
  rule for the docs pipeline diagram the agent is told to keep accurate.
- evals/agent-assets.lock.json: refreshed via
  `node evals/check-agent-drift.ts --update` since a tracked agent asset
  changed.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Nathan (nturinski) and others added 2 commits September 10, 2026 10:26
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The render-time label normalizer had rules for rect, cylinder, circle,
rhombus and edge labels, but none for the round shape `A(text)` — one of
the most common flowchart shapes. So a round node with a reserved
character still broke the whole diagram: mermaid v11 lexes the leading
`@` in `A(@azure/identity)` as edge-id / node-metadata syntax, and the
webview showed mermaid's syntax-error graphic rather than the plan.

Adds a round rule after the cylinder, circle and stadium rules so their
delimiters are already consumed by the time it runs. It skips a paren
body that a previous rule produced, which is what keeps stadium nodes
working: `A([Start])` has by then become `A(["Start"])`, and without the
guard the round rule would re-quote it into `A("[\"Start\"]")` and break
every stadium in the diagram.

Also documents that the parallelogram/trapezoid exclusion on the `[text]`
rule is a deliberate tradeoff — quoting those labels would flatten the
shape into a plain rectangle, silently changing diagrams that render
today — so a future reader doesn't undo it.

Verified against mermaid 11.17.0 under jsdom: all nine quotable shapes
with an `@`-leading label now parse after normalization, sixteen benign
and directive-bearing diagrams stay valid and idempotent, non-flowchart
diagrams are byte-identical, and the four repo debug-plan fixtures still
parse.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Nathan (nturinski) and others added 2 commits September 10, 2026 10:50
This reverts commit 773b038.

The round rule it added guarded with `/^[["/\\]/`, which only inspects the
first character of a label. Parentheses appearing *inside* a label body
therefore slipped past the guard and were re-quoted by the round rule in
the same chained pass, regressing cases that #1817 had correctly fixed:

  Web[Attendance Web<br/>(Vite dev server)]
    -> ["Attendance Web<br/>("Vite dev server")"]
  Web[Vite(5173)]
    -> ["Vite("5173")"]
  A -->|calls (async)| B
    -> |calls ("async")|

The first of those is the attendance fixture's own label in its unquoted
form, i.e. exactly what a debug plan written before #1813 looks like on
disk, so the regression would have reached real user files.

PR #1820 supersedes this commit with a correct fix: it runs the round
rule as a second pass over a re-split line, so parens inside a label are
already within a quoted segment and are structurally unreachable rather
than merely guarded against. That approach needs no guard at all and
additionally fixes a double-circle regression, `A(((Core)))`, that #1817
introduced. Reverting here so the two round-shape rules don't compete.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…iagrams

`quoteMermaidLabels` promises to be "a no-op on diagrams that already render",
but two shapes broke that contract and turned a diagram mermaid accepted into
one it rejected. Verified against mermaid 11.17.0 under jsdom.

Chained edge labels. `mapUnquotedSegments` splits a line on `"`, so an
already-quoted edge label leaves its own pipes in the *unquoted* pieces:
`A -->|"one"| B -->|"two"| C` yields a piece of `| B -->|`, whose two pipes the
edge rule read as a label and quoted into `|" B -->"|`. That is exactly the
shape this PR's own template rule asks the agent to emit. The edge rule now
anchors the opening `|` to the link that owns it, which also fixes the mixed
case where only some labels on a line are quoted.

Double circle. The circle rule's body accepts `(`, so `A(((Core)))` came out as
`A(("(Core")))`. A `(((text)))` rule now runs first and the circle rule skips a
body the double-circle rule already produced.

Both were also idempotency failures; the existing `is idempotent` test used a
single-edge line, so neither was reachable from it.

Render-time fallback. A regex pass over a real grammar can always meet a shape
it reads wrong, so `MermaidBlock` now retries with the diagram exactly as the
author wrote it when the rewrite is what mermaid rejects. Normalization can
still rescue a diagram mermaid would have refused, but it can no longer be the
reason a diagram that used to render stops rendering.

Validation: typecheck, eslint, `eval:drift`, `build:webviews`, 29/29 unit tests,
and every mermaid block in the repo's 6 plan/doc fixtures parses before and
after and is idempotent.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@nturinski

Copy link
Copy Markdown
Member Author

Copilot resolve the merge conflicts in this pull request

…-label-quoting-490

# Conflicts:
#	evals/agent-assets.lock.json
#	src/webviews/copilotOnRails/views/LocalPlanView.tsx

Co-authored-by: nturinski <5290572+nturinski@users.noreply.github.com>

Copilot AI commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Copilot resolve the merge conflicts in this pull request

Merged feat/CoR in (70a267b) and resolved both conflicts: LocalPlanView.tsx now keeps the onRenderStatus callback from feat/CoR while using the label-quoting renderMermaid fallback, and agent-assets.lock.json was regenerated via node evals/check-agent-drift.ts --update. tsc --noEmit, the webview build, and the drift check all pass clean.

Nathan (nturinski) and others added 2 commits September 14, 2026 13:05
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: nturinski <5290572+nturinski@users.noreply.github.com>
@nturinski
Nathan (nturinski) merged commit 1ccb471 into feat/CoR Sep 14, 2026
3 of 4 checks passed
@nturinski
Nathan (nturinski) deleted the nturinski-mermaid-label-quoting-490 branch September 14, 2026 20:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants