Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 29 additions & 7 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Product vision and roadmap

**Status:** Living document<br>
**Last reviewed:** 2026-09-01<br>
**Last reviewed:** 2026-09-06<br>
**Current stage:** Workflow parity and release (v0.9.0)

This document describes product direction, not fixed delivery dates. **Now** is
Expand Down Expand Up @@ -384,45 +384,57 @@ approved fresh retry timed out during authoring because the user-session
adapter's 120-second request limit could not use the 20-minute case budget. No
revised, approved, or exported artifact was produced. The private manual
baseline also targets a related but different role, which limits any future
comparison. The bounded provider-timeout, author-grounding, and corrective
comparison.

The bounded provider-timeout, author-grounding, and corrective
retry blockers #256, #258, and #260 are closed. The corrective observation
under issue #262 is closed as indeterminate: it used all three author attempts,
with output-token excess followed by two different single claim-text
factual-invariant failures. No draft reached the critic. The per-evidence
author-grounding guide blocker #264 then closed. The authorized observation
author-grounding guide blocker #264 then closed.

The authorized observation
under #266 repeatedly failed before author output because structured Claude
nonzero-exit errors were classified generically. Issue #267 delivered the
bounded classification fix. Keep #75 unvalidated; another authorized
comparison under #271 then reached all three author attempts. Output-token
excess was followed by three claim-text factual-invariant paths; the final
corrective attempt reduced them to one repeated path, but no proposal reached
the critic. Issue #272 delivered bounded exact-value citation completion: local
the critic.

Issue #272 delivered bounded exact-value citation completion: local
normalization may append only exact supporting retrieved chunks, while the
unchanged validator still rejects unsupported values. The authorized
post-completion observation under #275 then completed both provider roles on
their first attempts and reached the human boundary. Readiness remained false
with eight warnings and one unresolved chronology omission; the unsupported-
claim finding was a warning rather than a factuality error. No adjudication,
revision, approval, export, or submission occurred. Issue #277 stages the
revision, approval, export, or submission occurred.

Issue #277 stages the
candidate's exact adjudication through the shared application/local boundary,
and #278 preserves active provider-duration accounting across human waits and
restart boundaries. The fresh observation under #286 used 307 seconds of
active time, reached an accepted author draft on attempt three, and completed
independent critique on attempt one. Its twelve findings were materially
different from the previously confirmed nine, so no adjudication was applied.

Typed history then rejected the distinct new artifact because another run in
the workspace already held version 1. Migration 26 under #287 removed that
storage blocker. Under #290, the candidate then confirmed one accept, two
rejects, and nine nuanced decisions; the exact package persisted before any
provider call. The first revision attempt used an incorrect API-key default,
and the two authenticated user-session attempts retained the 120-second
per-request default and timed out. No revised artifact or new critique exists.

The fresh observation under #291 applied the explicit 20-minute request timeout
to every user-session subprocess. Authoring completed on attempt three and the
critic on attempt one. After the candidate confirmed two accepts, two rejects,
and six nuanced decisions, all three revision calls returned before timeout but
failed structured-response validation. The run exhausted after 959 active
seconds without a revised artifact or second critique. Issue #293 delivered
seconds without a revised artifact or second critique.

Issue #293 delivered
content-free failure-stage classification (`transport-parsing`,
`response-schema-validation`, `artifact-schema-validation`,
`factual-invariant-rejection`) and sanitized deterministic 10-finding carrier
Expand All @@ -431,11 +443,21 @@ exercised that classification during a fresh same-scope author attempt: all
three author calls returned before timeout but failed local factual-invariant
checks, correctly classified under `factual-invariant-rejection`. The run
exhausted after 313 seconds of active provider time without accepted author
output, so the critic was not called. The tenth observation under #298 also
output, so the critic was not called.

The tenth observation under #298 also
exhausted three author attempts across token-budget excess and
`factual-invariant-rejection` after 412 active seconds. Keep #75 unvalidated and
leave #250 blocked.

Issue #300 tightens numeric grounding in citation completion and author
validation: a number must match a whole source value, including its decimal
part, percentage marker, or magnitude suffix. For example, `2020` cannot
support a claimed metric of `20`. Deterministic regression coverage establishes
this bounded safety correction; it does not explain the live author failures
or change the indeterminate parity result. The next pilot observation remains
pending, and release preparation stays blocked on #75.

**Exit criterion:** The representative comparison records no factual-invariant
violations or unsupported model-added facts, preserves required sections and
chronology, meets the agreed relevance and coverage thresholds, and produces a
Expand Down
31 changes: 31 additions & 0 deletions packages/application/src/author-evidence-completion.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,37 @@ function singleClaimProposal(
}

describe("author evidence citation completion", () => {
it.each(["2020", "120", "20.5", "1,020", "20%", "20k", "20m", "20b"])(
"does not ground a metric of 20 with the different numeric value %s",
(sourceValue) => {
const evidence = [chunk("chunk-number", `delivered ${sourceValue} projects`)];
const uncited = singleClaimProposal("delivered 20 projects");
expect(completeAuthorEvidenceCitations(uncited, evidence)).toBe(uncited);

const cited = singleClaimProposal("delivered 20 projects", true, ["chunk-number"]);
expect(completeCvProposalIssues(cited, evidence)).toEqual([
expect.objectContaining({ code: "factual_invariant_violation" }),
]);
},
);

it.each([
["20", "delivered 20 projects."],
["20.5", "delivered 20.5 projects."],
["20%", "delivered 20% of projects."],
["20k", "delivered 20K projects."],
["2020", "delivered projects in 2020-2024."],
["20", "delivered 20 projects."],
])("retains exact numeric support for %s in %s", (value, sourceText) => {
const evidence = [chunk("chunk-number", sourceText)];
const completed = completeAuthorEvidenceCitations(
singleClaimProposal(`delivered ${value} projects`),
evidence,
);
expect(completed.sections[0]?.blocks[0]?.claims[0]?.evidenceChunkIds).toEqual(["chunk-number"]);
expect(completeCvProposalIssues(completed, evidence)).toEqual([]);
});

it("matches protected values with exact NFKC and en-US lowercase comparison", () => {
const text = "Staff Engineer";
const proposal = singleClaimProposal(text);
Expand Down
12 changes: 4 additions & 8 deletions packages/application/src/author-evidence-completion.ts
Original file line number Diff line number Diff line change
@@ -1,11 +1,7 @@
import type { ScoredEvidenceChunk } from "@draft-loop/domain";
import type { AuthorArtifactProposal } from "@draft-loop/schemas";

import { extractProtectedValues } from "./author-grounding.js";

function normalizedForComparison(value: string): string {
return value.normalize("NFKC").toLocaleLowerCase("en-US");
}
import { extractProtectedValues, supportsProtectedValue } from "./author-grounding.js";

function completedEvidenceChunkIds(
claim: AuthorArtifactProposal["sections"][number]["blocks"][number]["claims"][number],
Expand All @@ -25,10 +21,10 @@ function completedEvidenceChunkIds(
}

for (const chunk of retrievedEvidence) {
const supportsProtectedValue = protectedValues.some((value) =>
normalizedForComparison(chunk.text).includes(normalizedForComparison(value)),
const supportsAnyValue = protectedValues.some((value) =>
supportsProtectedValue(chunk.text, value),
);
if (!supportsProtectedValue || seen.has(chunk.id)) continue;
if (!supportsAnyValue || seen.has(chunk.id)) continue;
seen.add(chunk.id);
completedIds.push(chunk.id);
}
Expand Down
14 changes: 13 additions & 1 deletion packages/application/src/author-grounding.ts
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
import type { ScoredEvidenceChunk } from "@draft-loop/domain";

const protectedNumberPattern = /(?<![\p{L}\p{N}])\d+(?:[.,]\d+)*(?:%|[kmb])?(?![\p{L}\p{N}])/giu;

const protectedValuePatterns = [
/https?:\/\/[^\s)]+/giu,
/[\p{L}\p{N}._%+-]+@[\p{L}\p{N}.-]+\.[\p{L}]{2,}/gu,
/(?<![\p{L}\p{N}])\d+(?:[.,]\d+)*(?:%|[kmb])?(?![\p{L}\p{N}])/giu,
protectedNumberPattern,
/\b[\p{Lu}]{2,}(?:[+-][\p{Lu}\p{N}]+)*\b/gu,
/\b\p{Lu}[\p{L}'’-]+(?:\s+\p{Lu}[\p{L}'’-]+)+\b/gu,
/\b(?:at|for)\s+(\p{Lu}[\p{L}'’-]+)\b/gu,
Expand All @@ -25,6 +27,16 @@ function normalizedIdentity(value: string): string {
return value.normalize("NFKC").toLocaleLowerCase("en-US");
}

/** Match extracted numeric values as whole tokens, retaining their units. */
export function supportsProtectedValue(evidence: string, protectedValue: string): boolean {
const value = normalizedIdentity(protectedValue);
const source = normalizedIdentity(evidence);
if (/^\d/u.test(value)) {
return [...source.matchAll(protectedNumberPattern)].some((match) => match[0] === value);
}
return source.includes(value);
}

/** Extract exact protected values in first-occurrence order without duplicates. */
export function extractProtectedValues(value: string): readonly string[] {
const matches: ProtectedValueMatch[] = protectedValuePatterns.flatMap((pattern, patternIndex) =>
Expand Down
4 changes: 2 additions & 2 deletions packages/application/src/complete-cv.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import type { ScoredEvidenceChunk } from "@draft-loop/domain";
import type { AuthorArtifactProposal } from "@draft-loop/schemas";

import { extractProtectedValues } from "./author-grounding.js";
import { extractProtectedValues, supportsProtectedValue } from "./author-grounding.js";

export const factualInvariantIssueCodes = [
"missing_evidence",
Expand Down Expand Up @@ -68,7 +68,7 @@ export function completeCvProposalIssues(
});
}
for (const value of extractProtectedValues(claim.text)) {
if (!evidence.includes(normalized(value))) {
if (!supportsProtectedValue(evidence, value)) {
issues.push({
code: "factual_invariant_violation",
path: [...path, "text"],
Expand Down