Feature/knolo v4 phase5 trustbench

.github/workflows/ci.yml

@@ -16,3 +16,4 @@ jobs:
       - run: npm ci
       - run: npm run build
       - run: npm test
+      - run: npm run trustbench:test

.gitignore

@@ -66,6 +66,7 @@ Desktop.ini
 
 # Local rollout notes (not part of the public repository)
 LIVE_KBS_MVP.md
+/upcoming feature.md
 
 # ICP local harness generated seed data / runtime (also covered by tests/icp-local/.gitignore)
 tests/icp-local/data/

README.md

@@ -1,5 +1,9 @@
 # 🧠 Knolo
 
+**Current contract: Knolo v4 / Phase 5 TrustBench.** TypeScript `@knolo/core`
+builds v4 packs by default. v1–v3 packs remain readable; Python, Rust, and ICP
+are legacy compatibility profiles until their v4 conformance runners land.
+
 Knolo is a **local-first knowledge base engine** built around deterministic retrieval and portable `.knolo` packs.
 
 It provides:
@@ -20,35 +24,38 @@ Knolo prioritizes:
 * Portable binary knowledge packs
 * Optional ICP deployment for on-chain knowledge retrieval
 * Strict runtime contracts (optional advanced features)
+* Verifiable receipts, evidence spans, and deterministic retrieval-plan hashes
 
 > ⚠️ `knolo-core` (unscoped) on npm is deprecated. Use `@knolo/core`.
 
 ---
 
-# 📊 Retrieval Benchmark (March 2026)
+# 📊 TrustBench Reference (v4 / Phase 5)
 
-Knolo was evaluated using a deterministic lexical-first + optional rerank configuration.
+Knolo is evaluated using the checked-in deterministic lexical-first reference
+profile and the shared `conformance/` fixtures.
 
-**Run:** 2026-03-01
-**TopK:** 5
+**Contract:** `retrieval-v4.0`
+**Runner:** `npm run trustbench:test`
 
 ### Aggregate Metrics
 
 | Metric      | Score     |
 | ----------- | --------- |
-| Precision@5 | **0.490** |
-| Recall@5    | **1.000** |
-| MRR@5       | **0.867** |
-| nDCG@5      | **0.900** |
+| Metric | Definition |
+| --- | --- |
+| Recall@K | Relevant fixture sources retrieved |
+| MRR@K | Reciprocal rank of the first relevant source |
+| nDCG@K | Discounted ranking quality |
+| Abstention precision | Correct empty-scope decisions |
 
 ### Interpretation
 
-* ✅ **Recall@5 = 1.0** → All relevant documents were retrieved in every test query.
-* ✅ **High MRR (0.867)** → Relevant documents appear near the top.
-* ✅ **Strong nDCG (0.900)** → Ranking quality is consistently high.
-* 🔍 Precision reflects lexical grounding before rerank — by design, Knolo prioritizes deterministic recall over aggressive pruning.
+* Results are generated from committed fixtures, not a time-stamped external benchmark.
+* IDs, plan hashes, decisions, receipts, and corruption rejection are canonical.
+* Scores are rounded to six decimals for cross-runtime comparison.
 
-This benchmark demonstrates:
+This contract demonstrates:
 
 * Deterministic lexical retrieval is highly reliable.
 * Hybrid reranking improves ranking quality without sacrificing grounding.
@@ -212,12 +219,24 @@ cargo test
 
 ---
 
-# 🐍 Python Runtime Support (Phase 2)
+# 🐍 Python Runtime Support (Legacy compatibility profile)
 
 Knolo also ships a pure-Python runtime in `packages/core-python` for mounting existing `.knolo` packs and running deterministic lexical queries locally.
 
 It stays local-first, requires no vector database, and does not use embeddings on the default query path.
 
+The Python package currently reads v1–v3 packs and is not yet a v4 TrustBench-equivalent runtime.
+
+# 🧪 TrustBench / Conformance
+
+```bash
+npm run trustbench:generate
+npm run trustbench:test
+```
+
+The suite checks deterministic IDs, scores, plan hashes, receipts, Recall@K,
+MRR, nDCG, abstention precision, and fail-closed corruption handling.
+
 Install locally:
 
 ```bash

conformance/README.md

@@ -0,0 +1,36 @@
+# Knolo conformance fixtures
+
+This directory defines the shared Phase 5 contract. Fixtures are generated by
+`node trustbench/generate-fixtures.mjs` so binary artifacts are reproducible
+and do not depend on a developer's machine.
+
+```text
+conformance/
+  packs/{minimal-v1,standard-v3,verified-v4}.knolo
+  packs/corrupted/{truncated,section-digest}.knolo
+  queries/retrieval.jsonl
+  expected/retrieval-v4.0.json
+```
+
+Each query declares an ID, query text, options, and optional relevance labels.
+The canonical evaluator records hit IDs, scores rounded to six decimals,
+decision/rejection status, plan hashes, receipt verification, Recall@K, MRR,
+nDCG, and abstention precision. Exact scores are compared with a declared
+tolerance; IDs, decisions, plan hashes, and rejection behavior are exact.
+
+Supported runtime profiles are recorded in the output. A profile must not be
+marked equivalent until it consumes this same fixture set and canonicalizes
+the same result shape.
+
+## Commands
+
+From the repository root:
+
+```bash
+npm run trustbench:generate # rebuild packs and expected output
+npm run trustbench:test     # fail on result drift or corruption acceptance
+```
+
+The TypeScript profile is the reference output. Runtime adapters should add a
+profile runner under `trustbench/profiles/` only after they can consume the v4
+fixture and emit equivalent IDs, decisions, plan hashes, and receipt state.

conformance/expected/retrieval-v4.0.json

@@ -0,0 +1,125 @@
+{
+  "contract": "retrieval-v4.0",
+  "packVersion": 4,
+  "packFormat": "v4",
+  "aggregate": {
+    "recallAtK": 0.7,
+    "mrrAtK": 0.8,
+    "ndcgAtK": 0.922629,
+    "meanHitCount": 1,
+    "answerCount": 4,
+    "abstentionCount": 1,
+    "abstentionPrecision": 1,
+    "receiptsVerified": true
+  },
+  "rows": [
+    {
+      "id": "lexical-alpha",
+      "query": "alpha beta",
+      "hits": [
+        {
+          "id": "intro.md",
+          "blockId": 0,
+          "score": 1.156856
+        },
+        {
+          "id": "runtime.md",
+          "blockId": 1,
+          "score": 0.897493
+        }
+      ],
+      "decision": "answer",
+      "planHash": "sha256-a37a075b3b1fb36ed90154c679bdb78b47de9065fcebe302223206ecbfc617b2",
+      "receiptVerified": true,
+      "metrics": {
+        "recallAtK": 0.5,
+        "reciprocalRank": 1,
+        "ndcgAtK": 0.613147,
+        "hitCount": 2,
+        "relevantCount": 1
+      }
+    },
+    {
+      "id": "namespace-alpha",
+      "query": "alpha",
+      "hits": [
+        {
+          "id": "intro.md",
+          "blockId": 0,
+          "score": 1.269768
+        }
+      ],
+      "decision": "answer",
+      "planHash": "sha256-32f75441d37cffd6fd68d9bc6a76d67913163d121df50ab479d9218bc9ef64d5",
+      "receiptVerified": true,
+      "metrics": {
+        "recallAtK": 0.5,
+        "reciprocalRank": 1,
+        "ndcgAtK": 1,
+        "hitCount": 1,
+        "relevantCount": 1
+      }
+    },
+    {
+      "id": "source-runtime",
+      "query": "runtime",
+      "hits": [
+        {
+          "id": "runtime.md",
+          "blockId": 1,
+          "score": 2.56473
+        }
+      ],
+      "decision": "answer",
+      "planHash": "sha256-ff9b5b0daef232196dc01a57850d3f7d3bd8c953c5cece450f87c64d81875623",
+      "receiptVerified": true,
+      "metrics": {
+        "recallAtK": 1,
+        "reciprocalRank": 1,
+        "ndcgAtK": 1,
+        "hitCount": 1,
+        "relevantCount": 1
+      }
+    },
+    {
+      "id": "required-phrase",
+      "query": "alpha",
+      "hits": [
+        {
+          "id": "intro.md",
+          "blockId": 0,
+          "score": 1.641785
+        }
+      ],
+      "decision": "answer",
+      "planHash": "sha256-934edee9ecea91cdf3927e5654f0f9195548e906e1d806e4fe9cb213162dde3b",
+      "receiptVerified": true,
+      "metrics": {
+        "recallAtK": 0.5,
+        "reciprocalRank": 1,
+        "ndcgAtK": 1,
+        "hitCount": 1,
+        "relevantCount": 1
+      }
+    },
+    {
+      "id": "empty-scope",
+      "query": "alpha",
+      "hits": [],
+      "decision": "abstain",
+      "planHash": "sha256-2d5f95949ddcbbe4bc76c14b16019e4dd2a58234ac78df8d99986cc859a38cbd",
+      "receiptVerified": true,
+      "metrics": {
+        "recallAtK": 1,
+        "reciprocalRank": 0,
+        "ndcgAtK": 1,
+        "hitCount": 0,
+        "relevantCount": 0
+      }
+    }
+  ],
+  "runtime": {
+    "name": "typescript",
+    "version": "v20.8.0"
+  }
+}

conformance/queries/retrieval.jsonl

@@ -0,0 +1,5 @@
+{"id":"lexical-alpha","query":"alpha beta","options":{"topK":3},"relevant":["intro.md","other.md"]}
+{"id":"namespace-alpha","query":"alpha","options":{"topK":5,"namespace":"docs.alpha"},"relevant":["intro.md","other.md"]}
+{"id":"source-runtime","query":"runtime","options":{"topK":3,"source":"runtime.md"},"relevant":["runtime.md"]}
+{"id":"required-phrase","query":"alpha","options":{"topK":3,"requirePhrases":["alpha beta"]},"relevant":["intro.md","other.md"]}
+{"id":"empty-scope","query":"alpha","options":{"topK":3,"namespace":"does-not-exist"},"relevant":[]}

examples/icp-knowledge-canister/README.md

@@ -1,5 +1,9 @@
 # icp-knowledge-canister
 
+This example uses the legacy ICP v3 capability profile. TypeScript builds v4
+by default, so the example’s ICP pack builder intentionally requests v3 until
+the canister supports the v4 container and TrustBench contract.
+
 Minimal local `dfx` example for deploying the Knolo ICP canister without any middleware server.
 
 The example:

examples/langchain-basic/README.md

@@ -1,5 +1,9 @@
 # langchain-basic
 
+This adapter consumes the Knolo v4 TypeScript runtime and remains a thin
+retriever wrapper; receipts and retrieval plans are available from
+`@knolo/core`.
+
 Minimal LangChain-style retrieval with `@knolo/langchain` (no `langchain` dependency required).
 
 ```bash
@@ -12,4 +16,3 @@ npm run start
 ## License
 
 Apache-2.0
-

examples/llamaindex-basic/README.md

@@ -1,5 +1,9 @@
 # llamaindex-basic
 
+This adapter consumes the Knolo v4 TypeScript runtime and remains a thin
+retriever wrapper; receipts and retrieval plans are available from
+`@knolo/core`.
+
 Minimal LlamaIndex-style retrieval with `@knolo/llamaindex` (no `llamaindex` dependency required).
 
 ```bash
@@ -12,4 +16,3 @@ npm run start
 ## License
 
 Apache-2.0
-

examples/memory-overlay/README.md

@@ -1,5 +1,9 @@
 # Knolo Cortex memory overlay
 
+The overlay now serializes into v4 packs by default. Verify generated packs
+with `knolo verify` and use `queryWithReceipt()` when evidence/replay metadata
+is required.
+
 ```ts
 import {
   buildPack,

examples/nextjs-rag-chat/README.md

@@ -1,5 +1,9 @@
 # nextjs-rag-chat
 
+The scaffold targets Knolo v4 packs and the TypeScript TrustBench reference
+runtime. Use the CLI’s `--receipt` option when the application needs verifiable
+retrieval evidence.
+
 Use the published-style scaffold package from this monorepo:
 
 ```bash
@@ -20,4 +24,3 @@ npm run dev
 ## License
 
 Apache-2.0
-

examples/node-cli-rag/README.md

@@ -1,5 +1,8 @@
 # node-cli-rag
 
+This example targets the v4 CLI workflow: inspect, verify, migrate, and receipt
+output are available alongside normal deterministic queries.
+
 Minimal CLI retrieval flow using workspace `@knolo/cli`.
 
 ```bash
@@ -12,4 +15,3 @@ npm run build-pack && npm run query
 ## License
 
 Apache-2.0
-

package.json

@@ -13,6 +13,8 @@
     "test:rust": "cargo test --manifest-path packages/core-rust/Cargo.toml",
     "test:icp": "cargo test --manifest-path packages/core-rust/Cargo.toml && cargo test --manifest-path packages/icp-canister/Cargo.toml && bash scripts/check-icp-template-sync.sh",
     "test:icp:e2e": "bash scripts/e2e-icp-local.sh",
+    "trustbench:generate": "npm run build --workspace @knolo/core && node trustbench/generate-fixtures.mjs && node trustbench/run.mjs",
+    "trustbench:test": "npm run build --workspace @knolo/core && node trustbench/test.mjs",
     "icp:local": "bash tests/icp-local/run-local.sh"
   }
 }