Add closing-the-loop: signal-consuming agent pattern spec#15
Add closing-the-loop: signal-consuming agent pattern spec#15
Conversation
Defines the protocol for how a dedicated agent reads accumulated signals, detects recurring patterns via semantic clustering, and takes action (PRs, issues, reports) when patterns cross thresholds. Includes: - Pattern detection algorithm with volume-scaled thresholds - Severity classification (CRITICAL/HIGH/MEDIUM/LOW) - Known-patterns registry with full lifecycle - Partnership signal schema for consumer observability - Implementation checklist for teams adopting the pattern - End-to-end loop walkthrough Also updates README.md See Also section and CHANGELOG.md. Co-authored-by: Copilot <[email protected]>
There was a problem hiding this comment.
Pull request overview
Adds a new Agent Signals protocol document that specifies how a dedicated “signal-consuming” agent closes the loop by clustering accumulated signals into recurring patterns and triggering human-reviewed remediation actions (PRs/issues/reports). This completes the loop described in agent-signals/README.md by providing a concrete consumer-side contract.
Changes:
- Added
agent-signals/closing-the-loop.mddefining pattern detection, thresholds, severity, action sequences, and a known-patterns registry lifecycle. - Updated
agent-signals/README.mdto link to the new “Closing the Loop” spec from the “Close the loop” step and “See Also”. - Updated
CHANGELOG.mdwith a0.3.0entry documenting the addition and README change.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.
| File | Description |
|---|---|
| agent-signals/closing-the-loop.md | Introduces the consumer-agent specification (pattern detection → registry → action → verification), plus signal/registry examples. |
| agent-signals/README.md | Adds references to the new spec where the loop is described and in “See Also”. |
| CHANGELOG.md | Records the new spec and README linkage under v0.3.0. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| | Severity | Criteria | | ||
| |----------|----------| | ||
| | **CRITICAL** | Confidence ≤ 3 in 5+ signals for the same skill within the time window, OR escalation rate > 50% for any skill | |
There was a problem hiding this comment.
Severity criteria references "escalation rate > 50%" but the term "escalation rate" isn't defined (e.g., rate of escalation signals per run? per skill? per window?). Please define the metric precisely so different implementations classify severity consistently.
| | Severity | Criteria | | |
| |----------|----------| | |
| | **CRITICAL** | Confidence ≤ 3 in 5+ signals for the same skill within the time window, OR escalation rate > 50% for any skill | | |
| For severity classification, **escalation rate** is defined **per skill, within | |
| the current evaluation time window** as: | |
| `escalation rate = (# of signals for that skill in the window with escalation = true) / (total # of signals for that skill in the window)` | |
| In this table, **escalation rate > 50%** means **strictly more than half** | |
| of all signals for a single skill in the current time window explicitly | |
| request escalation. | |
| | Severity | Criteria | | |
| |----------|----------| | |
| | **CRITICAL** | Confidence ≤ 3 in 5+ signals for the same skill within the time window, OR escalation rate for a skill in the current time window is > 50% | |
| ### 1.1 Primary Input: Pattern Dimensions | ||
|
|
||
| The consuming agent reads the pattern dimensions from execution | ||
| signals (see [SIGNAL.md](SIGNAL.md)): | ||
|
|
||
| | Dimension | What It Tells the Consumer | | ||
| |-----------|---------------------------| | ||
| | `what_was_hard` | Friction points — where agents struggle | | ||
| | `skill_gap` | Missing capabilities agents wish they had | | ||
| | `tsg_gap` | Missing or inadequate guidance in documentation | | ||
| | `recurring_pattern` | Patterns the producing agent has seen before | | ||
| | `improvisation` | Novel approaches not covered by existing skills | | ||
| | `environment_blockers` | External issues (APIs, permissions, infra) | | ||
| | `what_worked` | Effective techniques worth preserving | |
There was a problem hiding this comment.
The spec lists environment_blockers as a pattern dimension, but agent-signals/SIGNAL.md's normative execution.patterns contract does not include this field. As written, a consumer following this spec won't be able to reliably parse signals that conform to the schema. Either add environment_blockers to the SIGNAL.md schema (and examples) or remove/mark it as an extension field here.
| ``` | ||
| For each pattern dimension d in [tsg_gap, skill_gap, what_was_hard, | ||
| environment_blockers, improvisation, recurring_pattern]: | ||
|
|
||
| 1. COLLECT all non-empty values of d from signals in the time window | ||
| 2. CLUSTER by semantic similarity (LLM-assisted grouping) | ||
| 3. COUNT occurrences per cluster, grouped by skill |
There was a problem hiding this comment.
This document says pattern detection covers "all seven dimensions" (§2.1 / checklist), but the detection algorithm (§2.2) iterates over only 6 dimensions and omits what_worked (and the thresholds table also doesn't define a threshold for it). Please reconcile: either include what_worked in the algorithm/thresholds, or explicitly state it is excluded from threshold-based detection and how it should be handled.
| ], | ||
|
|
||
| "action_taken": "pr_opened", | ||
| "action_ref": "https://github.com/org/repo/pull/123", |
There was a problem hiding this comment.
The partnership signal JSON example includes action_ref as a GitHub PR URL. agent-signals/SIGNAL.md's Privacy section says signals must never contain repository URLs; even in examples this conflicts with the protocol's stated constraints. Consider using a non-URL opaque identifier (e.g., pr_id/issue_id) or a clearly anonymized placeholder that isn't a repository URL.
| "action_ref": "https://github.com/org/repo/pull/123", | |
| "action_ref": "pr-123", |
| "pattern": "Multi-module rollback documentation missing", | ||
| "dimension": "tsg_gap", | ||
| "frequency": 4, | ||
| "severity": "medium", |
There was a problem hiding this comment.
Severity is defined earlier as CRITICAL/HIGH/MEDIUM/LOW (uppercase) and the registry schema uses uppercase, but this example uses lowercase "medium". Please standardize the severity enum casing across the doc and examples to avoid downstream parsing/validation mismatches.
| "severity": "medium", | |
| "severity": "MEDIUM", |
| dimension: tsg_gap | ||
| pattern_summary: "Multi-module rollback docs missing from remediation skill" | ||
| status: detected | acknowledged | in-progress | resolved | wont-fix | ||
| severity: CRITICAL | HIGH | MEDIUM | LOW | ||
| evidence: |
There was a problem hiding this comment.
In this YAML "schema" example, status: detected | acknowledged | ... and severity: CRITICAL | HIGH | ... look like literal scalar values rather than an enumeration. To avoid readers copying an invalid/ambiguous representation, consider formatting this as status: detected # one of: ... (same for severity) or listing allowed values separately.
- Define 'escalation rate' precisely (per skill, per window) - Mark environment_blockers as extension dimension (not in base SIGNAL.md schema) - Clarify what_worked is excluded from threshold detection (positive signal) - Fix severity casing: 'medium' → 'MEDIUM' for consistency - Replace repository URLs with opaque identifiers per SIGNAL.md privacy rules - Fix YAML enum formatting to avoid ambiguous pipe-separated values - Update checklist to reference correct section numbers Co-authored-by: Copilot <[email protected]>
What
Adds closing-the-loop.md to agent-signals/ — the spec for how a dedicated agent reads accumulated signals, detects recurring patterns, and takes action when patterns cross thresholds.
This is the missing piece the README references: 'Close the loop. Signals that are captured but never reviewed are noise.' Now there's a formal protocol for closing it.
What's in the spec
Also updated
Design decisions