fix(retain): derive document_id from content+context (idempotent re-ingest)

[kkroo]

What

When a caller supplies no explicit document_id, retain_batch minted a random uuid4. Because the id was random, re-ingesting byte-identical content created a duplicate document instead of upserting — the source of the exact-duplicate documents found in the BLO-9319 bank dedup (~14 exact dups).

This derives the fallback id deterministically from sanitized content namespaced by context (uuid5, preserving the 36-char id shape), so re-ingesting the same content+context upserts the same documents (id, bank_id) row via the existing ON CONFLICT path.

Why namespaced by context

Namespacing keeps two intentionally-distinct documents that happen to share identical content separate; only same-content and same-context re-ingests collapse. Content is sanitized with the same helper used for content_hash, so whitespace/encoding noise doesn't perturb the id.

Notes

• Forward-only: existing rows keep their random ids; no migration. Callers that pass an explicit document_id (e.g. the MCP agent_knowledge_ingest title-derived path) are unaffected.
• document_id is never parsed as a UUID downstream (verified), but is concatenated into chunk ids / rendered in the UI — hence keeping the uuid shape via uuid5.

Changes

• engine/retain/orchestrator.py: _derive_document_id() helper + fixed namespace constant; wired into both no-id fallback sites (main path + the multi-id batch-grouping path).
• tests/test_document_id_derivation.py: pure-function unit coverage (determinism, context namespacing, content sensitivity, uuid5 shape).
• tests/test_document_tracking.py: end-to-end idempotent-reingest regression (same content+context ⇒ 1 doc; different context ⇒ 2).

Verification

• Derivation algorithm proven via a stdlib replica (all 7 properties pass) + py_compile on all changed files.
• The package-level unit + integration tests run in CI (local run needs the full uv ML sync).

🤖 Generated with Claude Code

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.

Looks good. I did not find any high-confidence Critical or Important issues in this diff.

Suggestions (0)

None.

Strengths

• The fallback document id is deterministic while preserving the existing UUID-shaped id contract.
• The id derivation is scoped by sanitized context, which avoids collapsing same-content documents from intentionally different contexts.
• The regression coverage includes both pure helper properties and an end-to-end re-ingest case for the existing document upsert path.
• The namespace constant is guarded by a test, reducing the risk of accidentally orphaning future deterministic ids.

Recommended Action

1. Merge once CI passes, especially the package-level retain/document-tracking tests that exercise the async storage path.