Skip to content

fix(bundle): foundation's ROOT body was documentation prose sent as system instruction (A) + pin the tool-skills double-mount semantics (C) - #368

Merged
Brian Krabach (bkrabach) merged 2 commits into
mainfrom
lane/8rug-foundation-root-hygiene
Sep 6, 2026
Merged

Brian Krabach (bkrabach) merged 2 commits into
mainfrom
lane/8rug-foundation-root-hygiene

Conversation

@bkrabach

Copy link
Copy Markdown
Collaborator

What this fixes

(A) Foundation's ROOT bundle body was 499 characters of documentation prose — and that prose was the system instruction.

docs/BUNDLE_GUIDE.md:138:

Everything below the frontmatter is sent to the model as system instruction. It is not documentation, and it is not free.
...
Do not write a description of the bundle. That is what the frontmatter description: field is for — it is manifest metadata and is never sent to the model.

Root bundle.md carried a # Foundation Bundle v2.0 title, a "Key Features" table, a delegate-parameter snippet and an MCP config sample below the frontmatter — ahead of @foundation:context/shared/common-system-base.md. So every foundation-root session opened its system prompt with product documentation.

This is the same defect class as the notify-README body that app-cli #315 (dab0f7c) and #316 (1e47c38) fixed. This is that defect in the composition base every non-anchors bundle inherits.

The prose is not deleted — it moves into the frontmatter description:. The body becomes the single @mention it should always have been.

(C) The tool-skills double mount was measured before anything was touched. It is a finding, not a bug — no code changed. Details below.

(A) Fail-before / pass-after

Measured with the validator fixed in aaa5c47 — its extractor previously returned an empty string that satisfied every check, so these counts are honest only now.

check_body_is_instruction on bundle.md:

verdict prose_chars reason
main a0decc6 FLAG 499 no second-person address -- reads as documentation, not instruction
this branch OK 0 body carries no prose (mentions/scaffolding only)

Flagged preview on main:

This bundle provides the standard Amplifier foundation with the enhanced delegate tool for agent orchestration.
| Feature | Description |
|---------|-------------|
| **Delegate tool** | Two-parameter ...

Repo-wide body-instruction-check (same code the recipe step runs, validate-bundle-repo.yaml:4224):

main a0decc6 : {"files_checked": 32, "warnings": [{"type":"prose_below_frontmatter","file":"bundle.md","prose_chars":499}]}
this branch  : {"files_checked": 32, "warnings": []}

(A) Real-session check — foundation as ROOT

amplifier run "hi" --bundle foundation from a scratch dir carrying a project-scope .amplifier/settings.yaml source override (bundle.added.foundation: file://<worktree>, bundle.app: []). ~/.amplifier/cache was not edited.

llm:request.data.raw.system[0].text, head:

Before — session 5dfe9c07-d95d-41a8-815b-962c0ce5a0dd, total 130,055 chars:

# Foundation Bundle v2.0

This bundle provides the standard Amplifier foundation with the enhanced delegate tool for agent orchestration.

## Key Features

| Feature | Description |
|---------|-------------|
| **Delegate tool** | Two-parameter context control (depth + scope) |
...

After — session 268a6e56-d64d-4958-9ea4-2c2bdfd94a2e, total 129,178 chars:

@foundation:context/shared/common-system-base.md

---

<context_file paths="@foundation:context/shared/common-system-base.md → .../context/shared/common-system-base.md">
# Primary Core Instructions

You are Amplifier, an AI powered Microsoft CLI tool.
...

No feature table. The first content is common-system-base. (The bare @mention line itself remains — that is how mention expansion renders; the resolved file follows immediately.)

(C) Compose-semantics finding — tool-skills is mounted twice

Full write-up: docs/lanes/8rug-foundation-root-hygiene/C-compose-semantics-finding.md.

Measured first, per the spec. Verdict: nothing is silently dropped.

  • behaviors/agents.yaml:36-40 contributes config.skills: [amplifier-foundation#subdirectory=skills]
  • bundle.md:33amplifier-bundle-skills#subdirectory=behaviors/skills.yaml contributes config.skills: [amplifier-bundle-skills#subdirectory=skills] + config.visibility

Bundle.compose() (_dataclass.py:250) → merge_module_lists() keys by id or module, so the two collapse to one mount, and deep_merge() resolves the config: dicts recurse, lists concatenate with dedup, scalars are replaced by the later value. Effective composed config:

{"module":"tool-skills",
 "source":"...amplifier-bundle-skills@main#subdirectory=modules/tool-skills",
 "config":{"skills":["...amplifier-foundation@main#subdirectory=skills",
                     "...amplifier-bundle-skills@main#subdirectory=skills"],
           "visibility":{"enabled":true,...,"visibility_token_budget":5000}}}

Both orders keep both entries.

Why a test was still warranted. The double mount is safe only because of the list-concat rule added in 70d521f (#120) — a rule invisible at both declaration sites, with no failure signal if it regresses (the skills just stop being discoverable). tests/test_tool_skills_double_mount.py pins it. Disabling list-concat in deep_merge:

FAILED tests/test_tool_skills_double_mount.py::test_double_mount_keeps_both_skills_dirs_in_bundle_md_order
FAILED tests/test_tool_skills_double_mount.py::test_double_mount_keeps_both_skills_dirs_in_reverse_order
2 failed, 5 passed

Restored: 7 passed.

Residual risk, recorded not fixed: the two mounts are order-insensitive only because they name the same source. source is a scalar, so a future divergence means whichever composes last silently wins. test_both_mounts_agree_on_the_module_source turns that into a test failure instead of a session mystery.

(C) Real-session check — foundation's skills are discoverable

Same session 268a6e56, live hooks-skills-visibility list:

bundle-to-dot                  present=True   <- behaviors/agents.yaml mount (foundation skills dir)
creating-amplifier-modules     present=True   <- behaviors/agents.yaml mount (foundation skills dir)
per-repo-conventions           present=True   <- behaviors/agents.yaml mount (foundation skills dir)
image-vision                   present=True   <- bundle.md:33 skills.yaml mount (curated collection)

Both mounts reach the model.

Tests

2038 passed, 3 skipped, 2 failed in 21.46s

Both failures are pre-existing and reproduce unchanged on a0decc6 with this branch's changes stashed:

  • tests/test_sources.py::TestFileSourceHandler::test_resolve_existing_file
  • tests/test_grpc_adapter_main.py::TestVerifyModuleType::test_non_isinstance_object_with_mount_passes

Spend

$0.14 — two amplifier run "hi" sessions on haiku for the real-session checks ($0.07 each). No DTUs, no ledger rows.

Scope

This is PR 1 of 2 for model_performance-8rug. PR 2 carries (B) — the delegation-instructions.md split — and is gated on its own pre-registered eval. A and C are independent of that eval and land on verification.


Generated with Amplifier

Co-Authored-By: Amplifier 240397093+microsoft-amplifier@users.noreply.github.com

…ose, sent as system instruction

(A) Everything below a bundle's frontmatter is the system instruction whenever
that bundle is composed as the ROOT (docs/BUNDLE_GUIDE.md: "Everything below the
frontmatter is sent to the model as system instruction. It is not documentation,
and it is not free."). Root bundle.md carried a "# Foundation Bundle v2.0" title,
a feature table and an MCP config sample below the frontmatter -- so every
foundation-root session opened its system prompt with product documentation
rather than with common-system-base.

This is the same defect class as the notify-README body fixed by app-cli #315/#316.

Measured with the validator FIXED in aaa5c47 (whose extractor previously returned
an empty string that satisfied every check):

  main a0decc6 : verdict FLAG, prose_chars 499
                 "This bundle provides the standard Amplifier foundation with the
                  enhanced delegate tool for agent orchestration.\n| Feature | ..."
  this branch  : verdict OK,   prose_chars 0
                 "body carries no prose (mentions/scaffolding only)"
  repo-wide    : 1 prose_below_frontmatter WARNING -> 0, over 32 files checked

Real session, foundation as ROOT (project-scope source override; ~/.amplifier/cache
untouched), raw.system[0].text head:

  before (5dfe9c07, 130,055 chars): "# Foundation Bundle v2.0\n\nThis bundle
    provides the standard Amplifier foundation... | Feature | Description |..."
  after  (268a6e56, 129,178 chars): "@foundation:context/shared/common-system-base.md
    \n\n---\n\n<context_file ...># Primary Core Instructions\n\nYou are Amplifier..."

The prose is not deleted -- it moves into the frontmatter `description:`, which is
manifest metadata and is never sent to the model. The body is now the single
@mention it should always have been.

(C) tool-skills is mounted twice -- behaviors/agents.yaml (foundation's own skills
dir) and bundle.md:33 via amplifier-bundle-skills' skills.yaml. Measured first, per
the goal: NOTHING IS DROPPED. merge_module_lists collapses the two declarations by
module id and deep_merge concatenates list-typed config, so both skills sources
survive in both compose orders; a real foundation-root session lists all three
foundation skills alongside the curated collection. That is a finding, not a bug --
so no code changed. But it is safe only because of the list-concat rule added in
70d521f (#120), which is invisible at both declaration sites and fails SILENTLY if
it regresses. tests/test_tool_skills_double_mount.py pins it: disabling list-concat
fails 2 of its 7 tests; restored, 7 pass.

Suite: 2038 passed, 3 skipped, 2 failed -- both failures reproduce unchanged on
a0decc6 with these changes stashed (test_sources.py::test_resolve_existing_file,
test_grpc_adapter_main.py::test_non_isinstance_object_with_mount_passes).

Generated with Amplifier

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
…at the $10 authority

Terminal outcome: RESOLVED AT THE CAP (goal branch B). A and C are DONE and ship
in this PR. B's code half is DONE and measured on #369; B's eval is NOT-POSSIBLE
at $10 -- 12 valid runs / 0.67 observed validity = 18 launches x blended $4.08 =
$73.44, priced before spending, $0 spent on it, design not shrunk and relabelled.

Generated with Amplifier

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@bkrabach

Copy link
Copy Markdown
Collaborator Author

Manager verification — A and C both hold, each re-run by me in a scratch clone. Merging.

Head f2d7748, base a0decc6. CI 7/7 SUCCESS (ubuntu + Windows × 3.11/3.12/3.13). Suite 1971 passed, 2 failed — the known pre-existing pair.

(A) The prose is gone, and the real session proves it

main a0decc6 : {"files_checked": 32, "warnings":[{"type":"prose_below_frontmatter","file":"bundle.md","prose_chars":499}]}
this branch  : {"files_checked": 32, "warnings": []}

The fail-before is trustworthy only because aaa5c47 fixed the validator whose extractor returned an empty string that satisfied every check — before that, this warning could not be relied on. I confirmed the body myself: everything below the frontmatter is now the single @foundation:context/shared/common-system-base.md mention.

The real session is the check a unit test could not make:

before (5dfe9c07) after (268a6e56)
raw.system head # Foundation Bundle v2.0## Key Features| Feature | Description | @foundation:context/shared/common-system-base.md# Primary Core Instructions
total chars 130,055 129,178

No feature table. This is the notify-README defect one layer down — a bundle's documentation silently becoming the session's system prompt, exactly what app-cli #315/#316 fixed — and the same class the anchors family got weeks of scrutiny for while this base got none.

(C) Measured before fixing, and the answer was "no defect"

The spec required reading Bundle.compose()'s duplicate-module-id semantics before touching anything, because a double mount that merges correctly is a finding, not a bug. That is what it found: merge_module_lists() keys by id or module, so the two mounts collapse to one, and deep_merge() keeps both config.skills entries — in both orders.

I checked the load-bearing claim rather than take it: 70d521f is real and is exactly "fix: concatenate lists in deep_merge instead of replacing — fixes silent config loss in bundle composition (#120)". So the lane's framing is right — this double mount is safe only because of a rule invisible at both declaration sites.

And I confirmed the test's shape independently. It passes at a0decc6 and on the branch (7 passed both sides) — correct, since nothing is broken. The 2 failed, 5 passed in the PR body is a clearly-labelled counterfactual: revert 70d521f, watch the guard catch it, Restored: 7 passed. That is the right way to pin behaviour that is currently correct but only incidentally so, and the PR labels it as such rather than dressing it up as a fail-before against main.

The residual risk it recorded rather than fixed is the sharpest part: the two mounts are order-insensitive only because they name the same source, and source is a scalar — so a future divergence there would not concatenate. Reported, not silently absorbed.

On (B), for the record

Held as draft #369 with its arithmetic rather than shrunk to fit. That is the correct answer and it does not block A or C, exactly as the sequencing required.

@bkrabach
Brian Krabach (bkrabach) merged commit aac89ea into main Sep 6, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants