feat: scrape quality gates — classify outcome and roll back unusable indexes

ARCHITECTURE.md

@@ -248,6 +248,16 @@ The pipeline system manages asynchronous job processing with persistent state an
 
 Job states progress through: QUEUED → RUNNING → COMPLETED/FAILED/CANCELLED. All state transitions persist to database and emit events, enabling both recovery after restart and real-time monitoring. See [Event Bus Architecture](docs/concepts/eventbus-architecture.md) for detailed event flow diagrams.
 
+#### Quality Gates
+
+Before promoting a finished job to `COMPLETED`, `PipelineManager` runs a quality gate at a single seam (the spot that previously set `COMPLETED` unconditionally). The gate classifies the scrape from its stored metrics and, on a failing verdict, discards the staged version and marks the job `FAILED` with a typed `errorCode` — a **gate-then-rollback** flow:
+
+1. **Classify** — `evaluateOutcome()` (a pure function in `src/pipeline/outcomeGate.ts`) maps `JobOutcomeMetrics` (document count, distinct URLs, in-scope ratio, expect-terms match) to a `ScrapeOutcome` and optional `ScrapeErrorCode`. Order: empty → thin → off-topic → scope-drift → indexed.
+2. **Relevance signals** — when the caller supplies `expectTerms`, `src/pipeline/relevanceGate.ts` computes a ref-agnostic in-scope URL ratio (`computeInScopeRatio`, aligning a GitHub `/tree/` root with its `/blob/` children) and samples stored chunks for the expected terms (`sampleExpectTermsMatch`). These relevance axes are opt-in for this release; `denyPaths` exclusions apply to every scrape.
+3. **Rollback** — on a non-`indexed` verdict the manager calls `removeVersion()` to delete the half-indexed docs, then rejects job completion with a `QualityGateError` carrying the remediation hint. **Refresh and append jobs (`isRefresh` / `clean === false`) skip the gate entirely** so a transient thin re-index can never wipe pre-existing good docs.
+
+The resulting `outcome` and `errorCode` are surfaced on the public `PipelineJob` and exposed through `get_job_info` and `list_jobs`. Error codes: `EMPTY_RESULT`, `THIN_RESULT`, `OFF_TOPIC`, `SCOPE_DRIFT`, `LOCALE_REDIRECT_LOOP` (cyclic locale redirect detected in `HttpFetcher`), and `GITHUB_SUBPATH_NOT_FOUND` (a GitHub `/tree/` subpath matched no files).
+
 ### Content Processing
 
 Content processing follows a modular strategy-pipeline-splitter architecture:

README.md

@@ -155,6 +155,39 @@ See **[Embedding Models](docs/guides/embedding-models.md)** for configuring **Ol
 -   Web requests send `Accept: text/markdown, text/html;q=0.9, */*;q=0.8` by default. Servers that support Markdown content negotiation, including Cloudflare Markdown for Agents, can return Markdown directly so the scraper bypasses HTML-to-Markdown conversion for cleaner output.
 -   This behavior is automatic and requires no configuration. Custom `Accept` headers are preserved when provided.
 
+### Scrape Quality Gates
+
+A scrape is marked `completed` only when it indexed usable documentation. When a quality gate fails, the job is marked `failed`, the half-indexed version is rolled back (gate-then-rollback), and a machine-readable `errorCode` is reported through `get_job_info` / `list_jobs`.
+
+`scrape_docs` accepts these additional optional parameters:
+
+| Parameter | Type | Description |
+|---|---|---|
+| `expectTerms` | `string[]` | Terms expected to appear in the indexed docs. When set, sampled chunks are checked; an off-topic result fails with `OFF_TOPIC` (also enables scope-drift checks). |
+| `denyPaths` | `string[]` | Glob patterns (minimatch) excluded from indexing even when in scope. Defaults to `demos/**` and `examples/**` at any depth. |
+| `localeStrategy` | `"pin-en" \| "strip" \| "passthrough"` | Locale handling for fetched URLs. `pin-en` (default) sends `Accept-Language: en` and strips `hl`/`lang`/`locale` query params; `strip` only removes those params; `passthrough` leaves URL and headers unchanged. |
+
+Each finished job carries an `outcome` classification, and a failing gate also sets an `errorCode`:
+
+| `outcome` | Meaning |
+|---|---|
+| `indexed` | Useful documentation was indexed (success). |
+| `empty` | The crawl finished but indexed no content. |
+| `thin` | Fewer chunks than the minimum threshold were indexed. |
+| `degenerate` | Content was indexed but failed a relevance check (off-topic or scope drift). |
+| `failed` | The job failed before classification (scraper error). |
+
+`errorCode` values surfaced on failed jobs:
+
+-   `EMPTY_RESULT` — crawl indexed no content (check reachability, JS gating, redirect/anti-bot walls).
+-   `THIN_RESULT` — too few chunks indexed (widen `maxPages`/`maxDepth` or point at a richer docs root).
+-   `OFF_TOPIC` — indexed content did not contain the expected terms (pick a more specific URL).
+-   `SCOPE_DRIFT` — too few indexed URLs fall under the requested path (narrow the URL or set `denyPaths`).
+-   `LOCALE_REDIRECT_LOOP` — a cyclic locale redirect (e.g. `?hl=` ping-pong) was detected.
+-   `GITHUB_SUBPATH_NOT_FOUND` — a GitHub `/tree/` subpath matched no files (the error lists available top-level paths).
+
+The remediation hint for a failed gate is included in the job's `errorMessage`.
+
 ### Key Concepts & Architecture
 -   **[Deployment Modes](docs/infrastructure/deployment-modes.md)**: Standalone vs. Distributed (Docker Compose).
 -   **[Authentication](docs/infrastructure/authentication.md)**: Securing your server with OAuth2/OIDC.

src/mcp/mcpServer.ts

@@ -71,6 +71,20 @@ export function createMcpServerInstance(
           .boolean()
           .optional()
           .describe("Preserve hash fragments for hash-routed SPA documentation sites."),
+        expectTerms: z
+          .array(z.string().max(200))
+          .max(50)
+          .optional()
+          .describe("Terms expected in the docs; off-topic scrapes fail with OFF_TOPIC."),
+        denyPaths: z
+          .array(z.string().max(200))
+          .max(50)
+          .optional()
+          .describe("Glob paths excluded from indexing (default: demos/examples)."),
+        localeStrategy: z
+          .enum(["pin-en", "strip", "passthrough"])
+          .optional()
+          .describe("Locale handling for fetched URLs (default pin-en)."),
       },
       {
         title: "Scrape New Library Documentation",
@@ -86,6 +100,9 @@ export function createMcpServerInstance(
         scope,
         followRedirects,
         preserveHashes,
+        expectTerms,
+        denyPaths,
+        localeStrategy,
       }) => {
         // Track MCP tool usage
         telemetry.track(TelemetryEvent.TOOL_USED, {
@@ -113,6 +130,9 @@ export function createMcpServerInstance(
               scope,
               followRedirects,
               preserveHashes,
+              expectTerms,
+              denyPaths,
+              localeStrategy,
             },
           });
 

src/pipeline/PipelineManager.test.ts

@@ -23,7 +23,7 @@ import { type AppConfig, loadConfig } from "../utils/config";
 import { PipelineManager } from "./PipelineManager";
 import { PipelineWorker } from "./PipelineWorker";
 import type { InternalPipelineJob, PipelineJob, PipelineManagerCallbacks } from "./types";
-import { PipelineJobStatus } from "./types";
+import { PipelineJobStatus, ScrapeErrorCode, ScrapeOutcome } from "./types";
 
 // Mock dependencies
 vi.mock("../store/DocumentManagementService");
@@ -131,6 +131,16 @@ describe("PipelineManager", () => {
         id: 1,
         name: "test-lib",
       }),
+      // Quality-gate methods: default to a healthy result so existing happy-path
+      // jobs still transition to COMPLETED.
+      getVersionMetrics: vi.fn().mockResolvedValue({
+        documentCount: 10,
+        distinctUrls: 5,
+      }),
+      removeVersion: vi.fn().mockResolvedValue(undefined),
+      // Relevance-gate methods (only consulted when expectTerms is set).
+      sampleChunks: vi.fn().mockResolvedValue([]),
+      listIndexedUrls: vi.fn().mockResolvedValue([]),
     };
 
     // Mock the worker's executeJob method
@@ -269,6 +279,80 @@ describe("PipelineManager", () => {
     expect(jobAfter?.error?.message).toBe("fail");
   });
 
+  it("rolls back and fails the job when the gate returns empty", async () => {
+    // Arrange: a store whose getVersionMetrics reports zero docs after scrape.
+    (mockStore.getVersionMetrics as Mock).mockResolvedValue({
+      documentCount: 0,
+      distinctUrls: 0,
+    });
+    const removeSpy = mockStore.removeVersion as Mock;
+
+    const jobId = await manager.enqueueScrapeJob("lib", "1.0.0", {
+      url: "https://x/tree",
+      library: "lib",
+      version: "1.0.0",
+    });
+    await manager.start();
+    await vi.advanceTimersByTimeAsync(1);
+    await manager.waitForJobCompletion(jobId).catch(() => {}); // gate failure rejects
+
+    const job = await manager.getJob(jobId);
+    expect(job?.status).toBe(PipelineJobStatus.FAILED);
+    expect(job?.outcome).toBe(ScrapeOutcome.EMPTY);
+    expect(job?.errorCode).toBe(ScrapeErrorCode.EMPTY_RESULT);
+    expect(removeSpy).toHaveBeenCalledWith("lib", "1.0.0");
+  });
+
+  it("does NOT roll back a refresh job that fails the gate (no data loss)", async () => {
+    // A refresh preserves pre-existing good docs; a failing gate must never removeVersion.
+    (mockStore.getVersionMetrics as Mock).mockResolvedValue({
+      documentCount: 0,
+      distinctUrls: 0,
+    });
+    const removeSpy = mockStore.removeVersion as Mock;
+
+    const jobId = await manager.enqueueScrapeJob("lib", "1.0.0", {
+      url: "https://x/tree",
+      library: "lib",
+      version: "1.0.0",
+      isRefresh: true,
+    });
+    await manager.start();
+    await vi.advanceTimersByTimeAsync(1);
+    await manager.waitForJobCompletion(jobId).catch(() => {});
+
+    const job = await manager.getJob(jobId);
+    expect(removeSpy).not.toHaveBeenCalled();
+    expect(job?.outcome).toBe(ScrapeOutcome.INDEXED); // gate skipped for refresh
+  });
+
+  it("fails as OFF_TOPIC when expectTerms are absent from sampled chunks", async () => {
+    (mockStore.getVersionMetrics as Mock).mockResolvedValue({
+      documentCount: 50,
+      distinctUrls: 40,
+    });
+    (mockStore.sampleChunks as Mock).mockResolvedValue([
+      "list-it demo",
+      "mood-food demo",
+    ]);
+    (mockStore.listIndexedUrls as Mock).mockResolvedValue([]);
+    const removeSpy = mockStore.removeVersion as Mock;
+
+    const jobId = await manager.enqueueScrapeJob("gemini", "1.0.0", {
+      url: "https://github.com/google/generative-ai-docs",
+      library: "gemini",
+      version: "1.0.0",
+      expectTerms: ["generateContent"],
+    });
+    await manager.start();
+    await vi.advanceTimersByTimeAsync(1);
+    await manager.waitForJobCompletion(jobId).catch(() => {});
+
+    const job = await manager.getJob(jobId);
+    expect(job?.errorCode).toBe(ScrapeErrorCode.OFF_TOPIC);
+    expect(removeSpy).toHaveBeenCalled();
+  });
+
   it("should cancel a job via cancelJob API", async () => {
     let resolveJob: () => void = () => {};
     mockWorkerInstance.executeJob.mockReturnValue(

src/pipeline/PipelineManager.ts

@@ -16,12 +16,15 @@ import { ScrapeMode } from "../scraper/types";
 import type { DocumentManagementService } from "../store";
 import { VersionStatus } from "../store/types";
 import type { AppConfig } from "../utils/config";
+import { ScraperError } from "../utils/errors";
 import { logger } from "../utils/logger";
-import { CancellationError, PipelineStateError } from "./errors";
+import { CancellationError, PipelineStateError, QualityGateError } from "./errors";
+import { evaluateOutcome, type OutcomeVerdict } from "./outcomeGate";
 import { PipelineWorker } from "./PipelineWorker"; // Import the worker
+import { computeInScopeRatio, sampleExpectTermsMatch } from "./relevanceGate";
 import type { IPipeline } from "./trpc/interfaces";
-import type { InternalPipelineJob, PipelineJob } from "./types";
-import { PipelineJobStatus } from "./types";
+import type { InternalPipelineJob, JobOutcomeMetrics, PipelineJob } from "./types";
+import { PipelineJobStatus, type ScrapeErrorCode, ScrapeOutcome } from "./types";
 
 /**
  * Manages a queue of document processing jobs, controlling concurrency and tracking progress.
@@ -83,6 +86,8 @@ export class PipelineManager implements IPipeline {
       updatedAt: job.updatedAt,
       sourceUrl: job.sourceUrl,
       scraperOptions: job.scraperOptions,
+      outcome: job.outcome,
+      errorCode: job.errorCode,
     };
   }
 
@@ -629,6 +634,60 @@ export class PipelineManager implements IPipeline {
     }
   }
 
+  /**
+   * Runs quality gates on a finished job. On a failing verdict, discards the staged
+   * version (gate-then-rollback) and returns the verdict; otherwise returns null.
+   *
+   * @param job - The finished internal job to evaluate.
+   * @returns The failing verdict (after rolling back), or null when the job passed.
+   */
+  private async applyQualityGate(
+    job: InternalPipelineJob,
+  ): Promise<OutcomeVerdict | null> {
+    // SAFETY (review F2): never gate-rollback a refresh or append (clean=false). Those
+    // flows preserve a version that may already hold good docs; removeVersion() would
+    // destroy them (data loss). Refresh only updates changed pages, so a transient thin
+    // re-index must not wipe the prior index. Treat as indexed and skip the gate.
+    if (job.scraperOptions.isRefresh || job.scraperOptions.clean === false) {
+      job.outcome = ScrapeOutcome.INDEXED;
+      return null;
+    }
+    const counts = await this.store.getVersionMetrics(job.library, job.version);
+    const opts = job.scraperOptions;
+    let inScopeUrlRatio: number | undefined;
+    let expectTermsMatched: boolean | undefined;
+    // Review F6: scope-drift and off-topic are OPT-IN for v1 (rollout is hard-fail-immediate).
+    // Both relevance axes are computed only when the caller signals intent via `expectTerms`.
+    // denyPaths still trims demos/examples for everyone; a future release can promote
+    // scope-drift to default-on once warn-mode telemetry confirms low false-positive rates.
+    if (opts.expectTerms?.length) {
+      const urls = await this.store.listIndexedUrls(job.library, job.version);
+      inScopeUrlRatio = urls.length ? computeInScopeRatio(opts.url, urls) : undefined;
+      const chunks = await this.store.sampleChunks(job.library, job.version, 20);
+      expectTermsMatched = sampleExpectTermsMatch(chunks, opts.expectTerms);
+    }
+    const metrics: JobOutcomeMetrics = {
+      documentCount: counts.documentCount,
+      distinctUrls: counts.distinctUrls,
+      pagesScraped: job.progressPages ?? 0,
+      inScopeUrlRatio,
+      expectTermsMatched,
+    };
+    const verdict = evaluateOutcome(metrics);
+    if (verdict.outcome === ScrapeOutcome.INDEXED) {
+      job.outcome = ScrapeOutcome.INDEXED;
+      return null;
+    }
+    logger.warn(
+      `❌ Quality gate failed for ${job.library}@${job.version}: ` +
+        `${verdict.outcome} (${verdict.errorCode}) — discarding staged docs`,
+    );
+    await this.store.removeVersion(job.library, job.version);
+    job.outcome = verdict.outcome;
+    job.errorCode = verdict.errorCode;
+    return verdict;
+  }
+
   /**
    * Executes a single pipeline job by delegating to a PipelineWorker.
    * Handles final status updates and promise resolution/rejection.
@@ -662,6 +721,23 @@ export class PipelineManager implements IPipeline {
         throw new CancellationError("Job cancelled just before completion");
       }
 
+      // Quality gate: only promote to COMPLETED if useful docs were indexed.
+      const gateFailure = await this.applyQualityGate(job);
+      if (gateFailure) {
+        await this.updateJobStatus(
+          job,
+          PipelineJobStatus.FAILED,
+          gateFailure.remediation,
+        );
+        job.errorCode = gateFailure.errorCode;
+        job.finishedAt = new Date();
+        const err = new QualityGateError(gateFailure);
+        job.error = err;
+        logger.info(`❌ Job failed quality gate: ${jobId}: ${gateFailure.errorCode}`);
+        job.rejectCompletion(err);
+        return;
+      }
+
       // Mark as completed
       await this.updateJobStatus(job, PipelineJobStatus.COMPLETED);
       job.finishedAt = new Date();
@@ -684,6 +760,9 @@ export class PipelineManager implements IPipeline {
       } else {
         // Handle other errors
         const errorMessage = error instanceof Error ? error.message : String(error);
+        if (error instanceof ScraperError && error.code) {
+          job.errorCode = error.code as ScrapeErrorCode;
+        }
         await this.updateJobStatus(job, PipelineJobStatus.FAILED, errorMessage);
         job.error = error instanceof Error ? error : new Error(String(error));
         job.finishedAt = new Date();

src/pipeline/errors.ts

@@ -1,3 +1,5 @@
+import type { OutcomeVerdict } from "./outcomeGate";
+
 export class PipelineError extends Error {
   constructor(
     message: string,
@@ -13,6 +15,17 @@ export class PipelineError extends Error {
 
 export class PipelineStateError extends PipelineError {}
 
+/**
+ * Raised when a scrape finished but failed a quality gate (empty/thin/degenerate).
+ * Carries the full verdict (outcome, errorCode, remediation) for callers.
+ */
+export class QualityGateError extends Error {
+  constructor(public readonly verdict: OutcomeVerdict) {
+    super(verdict.remediation ?? `Quality gate failed: ${verdict.outcome}`);
+    this.name = "QualityGateError";
+  }
+}
+
 /**
  * Error indicating that an operation was cancelled.
  */