Skip to content

fix: add observability consistently across NFR stages#404

Open
jeromevdl wants to merge 1 commit into
awslabs:v2from
jeromevdl:feat/observability-alignment
Open

fix: add observability consistently across NFR stages#404
jeromevdl wants to merge 1 commit into
awslabs:v2from
jeromevdl:feat/observability-alignment

Conversation

@jeromevdl

Copy link
Copy Markdown

Summary

Fixes #398 — NFR categories were inconsistent across stages: nfr-requirements.md listed 5 categories (Performance, Security, Scalability, Reliability, Observability) while requirements-analysis.md, nfr-design.md, and infrastructure-design.md only listed 4 (missing Observability).

Changes

Core stage files (core/aidlc-common/stages/)

  • inception/requirements-analysis.md: Added "observability" to the NFR category list in Step 6 (completeness analysis) and Step 10 (generate requirements)
  • construction/nfr-requirements.md: Added observability-requirements artifact to produces:, outputs:, and Step 6; updated condition: to list all 5 NFR categories
  • construction/nfr-design.md: Added observability-design artifact to produces:, outputs:, Step 5, and Step 6; added observability-requirements to consumes:; updated upstream-coverage sensor list
  • construction/infrastructure-design.md: Added observability-design to consumes: list and upstream-coverage sensor description

Generated dist/

  • Regenerated all 4 harness trees via bun scripts/package.ts (pass --check)

Tests

  • Regenerated tests/fixtures/designer-export/export.json golden fixture
  • Updated tests/integration/t66.test.ts element count assertion (122→124 artifacts, 11→13 agents) and test label

Artifact flow (after fix)

requirements-analysis (observability in NFR dimensions)
    → nfr-requirements (produces: observability-requirements)
        → nfr-design (consumes: observability-requirements, produces: observability-design)
            → infrastructure-design (consumes: observability-design, produces: monitoring-design)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of the project license.

@jeromevdl jeromevdl force-pushed the feat/observability-alignment branch from 0249166 to e089615 Compare June 19, 2026 09:42
@jeromevdl jeromevdl force-pushed the feat/observability-alignment branch from e089615 to 6ee6795 Compare June 29, 2026 07:25
Add observability as a first-class NFR category across all stages that
enumerate NFR dimensions, fixing the inconsistency where only
nfr-requirements.md listed it while requirements-analysis.md,
nfr-design.md, and infrastructure-design.md omitted it.

Changes:
- requirements-analysis.md: add observability to Step 6 and Step 10
- nfr-requirements.md: add observability-requirements artifact (produces,
  outputs, Step 6), update condition description
- nfr-design.md: add observability-design artifact (produces, consumes,
  outputs, Step 5, Step 6), add observability-requirements to consumes
- infrastructure-design.md: add observability-design to consumes list
- Regenerate dist/ for all harnesses
- Update golden fixture and t66 element count assertion

Closes awslabs#398

@apackeer apackeer left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix; the shape is right and the artifact graph is internally consistent. Every new produces has a consumer, and every new required consumes has a producer in every scope that runs the consumer stage. Requesting changes for rebase and same-commit hygiene the current v2 requires.

Blockers

  1. Rebase required. Base is 2.1.3 (c95070e); v2 is now 2.2.0 (b67798c) with six intervening commits (2.1.4 - 2.2.0). git merge-tree origin/v2 refs/pull/404/head reports a hard conflict on tests/integration/t66.test.ts; the stage .md files and dist trees auto-merge textually but every dist/*/tools/data/stage-graph.json and tests/fixtures/designer-export/export.json needs a real regeneration on the rebased tree.

  2. Version trio missing. core/tools/aidlc-version.ts still reads 2.1.3, and there is no matching heading in CHANGELOG.md or update to the README badge. AGENTS.md Changelog Policy requires all three in the same commit and tests/unit/t68-version-changelog-sync.test.ts enforces it. Versions 2.2.1 - 2.2.13 are already claimed by open PRs, so 2.2.14 is the next free slot (re-bump if a sibling merges first).

  3. t66 agent count stale. The PR sets artifacts=123, agents=13 in tests/integration/t66.test.ts; v2 has artifacts=122, agents=14 after the composer agent landed in 2.2.0. On rebase the target is stages=32, scopes=9, artifacts=123, agents=14, with both expect(...).toBe(...) lines matching. Your artifact math (net +1: two added, one removed) is correct.

  4. Golden fixture must be re-exported after rebase. tests/fixtures/designer-export/export.json was regenerated at 2.1.3; the byte-exact assertion in t66 needs a fresh export on the rebased tree so it includes the composer agent plus the observability additions.

  5. Docs still name monitoring-design.md. Six references the PR removes from core/ but leaves in the shipped docs:

    • docs/reference/03-orchestrator.md:565
    • docs/reference/04-stages/construction.md:450, :515, :538, :550, :1044
      AGENTS.md Documentation Policy asks for these to update in the same commit. While there, add rows for observability-requirements.md and observability-design.md wherever the docs enumerate produced artifacts by name.
  6. infrastructure-design.md content preservation. Step 5 drops the "Monitoring and Observability" bullet (four areas becomes three) and Step 6 drops the monitoring-design.md bullet. The short "observability tooling" phrase added into Infrastructure Services does not fully preserve the operational-visibility detail (metrics platform, log aggregation, tracing backend, alerting) that lived in the removed bullets. Please either expand the Infrastructure Services phrasing or reference observability-design.md explicitly, so the reader still sees where that content lives.

Non-blocking

  1. The PR summary claims (122 -> 124 artifacts, 11 -> 13 agents). The actual diff is 122 -> 123, and agents is untouched by the diff. Worth fixing in the PR body so reviewers do not reconcile silently.

  2. core/knowledge/aidlc-quality-agent/nfr-reliability-guide.md is titled "NFR Reliability and Observability Guide", which pre-dates promoting observability to a peer NFR category. Optional pass to reshape the knowledge companions so observability reads like a peer rather than a sub-topic of reliability. Not a merge blocker.

  3. Cross-PR interaction with #402 (same author, also touches nfr-design.md + infrastructure-design.md). Whichever merges second will re-apply on top of the first. Heads-up only.

  4. Post-rebase, run t66 and confirm the advisory-edge baseline still holds. Adding required consumes changes the dangling calculation for scopes that skip the producer stage; a diff there is a design question, not a mechanical pin update.

Design confirmation (for the record)

The observability chain closes cleanly in scopes that run every stage from nfr-requirements through infrastructure-design. nfr-requirements produces observability-requirements; nfr-design consumes it and produces observability-design; infrastructure-design and observability-setup consume observability-design. Scopes that skip nfr-design do not consume observability-design either, and the dangling-consume machinery from 2.1.7 covers the case where nfr-requirements produces observability-requirements without a downstream consumer in scope.

Ping me post-rebase and I will re-review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants