sync(hindsight): merge upstream vectorize-io 0.8.3 (resolves #12 conflicts)

.claude/skills/code-review/SKILL.md

@@ -172,7 +172,8 @@ For each non-trivial change:
 If any files in `hindsight-integrations/` were added or changed, verify:
 - **Tests exist** — the integration must have tests that simulate/exercise the external framework (not just pure unit tests of helpers). Check for a `tests/` directory with meaningful test files.
 - **CI job exists** — check `.github/workflows/test.yml` for a corresponding `test-<name>-integration` job. If missing, flag it.
-- **Release process** — check that the integration name is in the `VALID_INTEGRATIONS` array in `scripts/release-integration.sh`. If missing, flag it.
+- **Release process** — check that the integration name is in the `VALID_INTEGRATIONS` array in `scripts/release-integration.sh` AND in the `INTEGRATIONS` dict in `hindsight-dev/hindsight_dev/generate_changelog.py` (the changelog generator keeps its own list; a release fails at the changelog step if the name is missing there). If either is missing, flag it.
+- **Docs gallery + sidebar entry** — the integration must have an entry in `hindsight-docs/src/data/integrations.json`. This file is the **single source of truth** that drives both the integrations gallery and the docs sidebar (the sidebar category is injected from it at render time across all docs versions). The entry needs an internal `/sdks/integrations/<slug>` `link` and a matching page at `hindsight-docs/docs-integrations/<slug>.md(x)`. The `hindsight-docs/scripts/check-integrations.mjs` build step enforces both directions — forward: every internal JSON entry has a doc page; reverse: every released tag (`integrations/<name>/vX.Y.Z`) appears in the JSON (private infra like `cloudflare-oauth-proxy` is in the script's `EXCLUDED` set). Flag any integration that is released (or being released) but missing from `integrations.json`, and any JSON entry without a doc page. Do **not** hand-edit `versioned_sidebars/*.json` to add integration links — they are positional placeholders filled from the JSON.
 - **Code standards** — the integration code must follow all Python style rules (type hints, no raw dicts, no tuple returns, etc.).
 
 ### 10. Check MCP tool registration completeness
@@ -191,6 +192,18 @@ If a migration adds a new PostgreSQL table (look for `CREATE TABLE` / `op.create
 - The guard test `test_backup_tables_covers_entire_schema` in `tests/test_admin_backup_restore.py` enforces this — flag it as a **must fix** if a new table is absent from `BACKUP_TABLES`.
 - Oracle-only tables (e.g. `observation_sources`) are intentionally excluded — admin backup/restore is PostgreSQL-only.
 
+### 11b. Check new config flags update the env template
+
+If the diff adds a new configuration field (a new `ENV_*` / `HINDSIGHT_*` env var
+in `hindsight-api-slim/hindsight_api/config.py`):
+- **`.env.example`** (repo root) — must add the variable (commented if optional)
+  alongside the docs entry in `hindsight-docs/docs/developer/configuration.md`.
+  A flag added to `config.py` but absent from `.env.example` is a **should fix**.
+- **`hindsight-embed/hindsight_embed/env.example`** — the bundled copy must stay
+  byte-identical to the repo-root `.env.example` (it seeds embed/profile configs).
+  The `test_bundled_template_matches_repo_root` sync test fails on drift; if the
+  root file changed without re-copying, flag it as a **must fix**.
+
 ### 12. Review against other coding standards
 
 Check the diff for violations of the standards listed above:
@@ -217,6 +230,7 @@ Present a clear summary organized by severity:
 - Direct DB access (raw SQL / `acquire_with_retry` / `fq_table`) in an `api/` handler instead of a `MemoryEngine` method
 - Tenant-scoped data accessed without authentication enforced in the engine (`_authenticate_tenant` / `get_bank_profile`)
 - New integration missing tests, CI job, or release-integration.sh entry
+- Released/added integration missing from `hindsight-docs/src/data/integrations.json`, or a JSON entry with no `docs-integrations/<slug>` page (fails the docs build via `check-integrations.mjs`)
 - New PostgreSQL table missing from `BACKUP_TABLES` in `admin/cli.py` (silent data loss on restore)
 
 **Should fix** — issues that hurt code quality:

.claude/skills/hs-release/SKILL.md

@@ -0,0 +1,116 @@
+---
+name: hs-release
+description: Cut a core Hindsight release (vX.Y.Z) and open the changelog + blog PR. Use when asked to cut/start a release, bump the version, or publish a new Hindsight version.
+user_invocable: true
+---
+
+# Hindsight Release
+
+Cut a **core** Hindsight release and open the accompanying changelog/blog PR. This is for the core
+product version (API, clients, CLI, control plane, Helm). **Integrations are versioned
+independently** — use `scripts/release-integration.sh` for those, not this skill.
+
+The release is **irreversible and outward-facing**: it tags a version and pushes it straight to
+`main`, which triggers CI that publishes packages to PyPI / npm / Helm. Confirm the version number
+and that the intended fixes are already merged to `main` before you start.
+
+## Step 0 — Pre-flight
+
+1. **Decide the base.** A release is cut from the latest `origin/main`, never from a feature
+   branch. `git fetch origin --tags` first. Confirm the "couple of fixes" the user means are
+   actually merged to `main` (`git log v<prev>..origin/main --oneline`).
+2. **Find where `main` is checked out.** `main` is often already checked out in a sibling worktree
+   (`git worktree list`). You **cannot** check out `main` in a second worktree — run the release in
+   the worktree that already holds it. If that worktree is dirty with throwaway cruft
+   (`.next-*` tsconfig paths, screenshots), `git stash push -u`, fast-forward to `origin/main`,
+   run the release, then `git stash pop`.
+3. **Pitfall:** never pipe the checkout in an `&&` chain like
+   `git checkout main 2>&1 | tail && git reset --hard ...` — the pipe's exit status is `tail`'s
+   (always 0), so a failed checkout won't stop the chain and the `reset` fires on the **wrong
+   branch**. Check out as its own command and verify `git branch --show-current` before resetting.
+
+## Step 1 — Cut the release
+
+Run from the worktree on a clean `main`:
+
+```bash
+./scripts/release.sh <version>     # e.g. 0.8.1  (no leading v)
+```
+
+`release.sh` bumps the version in every component, regenerates the OpenAPI spec + all client SDKs,
+updates docs versioning, commits `Release v<version>`, tags `v<version>`, and **pushes the commit
+and tag directly to `main`**. The push triggers the `Release` GitHub Actions workflow that builds
+and publishes the packages. It is **not** a PR.
+
+Verify after: `gh run list --limit 5` should show the `Release v<version>` workflow running, and
+`git ls-remote --tags origin v<version>` should return the tag.
+
+## Step 2 — Changelog + blog PR (separate)
+
+Done **after** the tag exists, as its own PR (precedent: v0.8.0 = #2053, v0.8.1 = #2080). Work on a
+branch off the new `main`:
+
+```bash
+git checkout -b docs-changelog-<version> origin/main
+```
+
+Only spin up a separate worktree (`git worktree add ../hindsight-changelog-<version> -b
+docs-changelog-<version> origin/main`) if you can't get a clean checkout otherwise — e.g. `main` is
+held in another worktree and the current one has work you don't want to disturb.
+
+**Branch naming:** use the `docs-` (hyphen) convention, e.g. `docs-changelog-0.8.1`. A remote
+branch literally named `docs` exists, so any `docs/...` branch is rejected on push with
+`directory file conflict`.
+
+### Changelog
+
+```bash
+uv run --directory hindsight-dev generate-changelog <version>
+```
+
+LLM-summarizes the commits between the previous tag and `v<version>` and prepends an entry to
+`hindsight-docs/src/pages/changelog/index.md`. Requires `OPENAI_API_KEY` (already in the repo
+`.env`). It excludes `hindsight-integrations/` source, but new integrations whose commits also
+touched docs will still appear — that matches precedent, leave them in the **changelog**.
+
+### Blog post
+
+Hand-write `hindsight-docs/blog/YYYY-MM-DD-version-X-Y-Z.md` (mirror an existing one; patch
+releases are short — see `2026-06-02-version-0-7-2.md`). Guidance:
+
+- **Explain user impact, not internals/mechanism.** Lead with what the user can now do and what to
+  set. Config/env-var names are fine (developer-facing), code symbols and internals are not.
+- **Do not list integrations in the release blog.** The core blog covers core engine / API /
+  ops changes; each integration ships its own changelog. (Integrations may still appear in the
+  generated `changelog/index.md` — that's fine; just keep them out of the blog.)
+- Call out an upgrade recommendation when there are operational/data-integrity fixes.
+- Validate formatting: `npx prettier --check <blog file>`.
+
+### Sync the docs skill
+
+```bash
+./scripts/generate-docs-skill.sh
+```
+
+Refreshes `skills/hindsight-docs/references/changelog/index.md`. It will also bump
+`skills/hindsight-docs/references/openapi.json` by one version — `release.sh` regenerates the skill
+*before* bumping OpenAPI, so the skill copy lags a version in the release commit; this step syncs
+it. Expect a one-line `version` diff there; keep it.
+
+### Commit, push, PR
+
+```bash
+git add -A
+git commit --no-verify -m "docs: changelog and blog post for v<version>"
+git push -u origin docs-changelog-<version>
+gh pr create --base main --title "docs: changelog and blog post for v<version>" --body "..."
+```
+
+Expected files in the PR: the changelog entry, the new blog post, the regenerated skill changelog
+mirror, and the skill `openapi.json` version sync.
+
+## Cleanup
+
+If you created a temporary worktree, remove it once the PR is up
+(`git worktree remove ../hindsight-changelog-<version>`; the branch stays on origin). Restore any
+stash you popped in Step 0.

.env.example

@@ -47,6 +47,13 @@ HINDSIGHT_API_LLM_BASE_URL=https://api.openai.com/v1
 HINDSIGHT_API_HOST=0.0.0.0
 HINDSIGHT_API_PORT=8888
 HINDSIGHT_API_LOG_LEVEL=info
+# Optional retain chunking override for structured logs/transcripts.
+# Unset uses HINDSIGHT_API_RETAIN_CHUNK_SIZE as the structured-chunk limit.
+# HINDSIGHT_API_RETAIN_STRUCTURED_CHUNK_SIZE=
+
+# Dry-run extraction preview endpoint (POST /memories/dry-run-extract). Enabled by default; it makes
+# a real LLM call but stores nothing. Set to false to remove the endpoint (returns 404).
+# HINDSIGHT_API_ENABLE_DRY_RUN_EXTRACT=true
 
 # Base Path / Reverse Proxy Support (Optional)
 # Set these when deploying behind a reverse proxy with path-based routing
@@ -59,6 +66,7 @@ HINDSIGHT_API_LOG_LEVEL=info
 # HINDSIGHT_API_READ_DATABASE_URL=  # Optional read-replica URL. When set, recall queries (semantic, BM25, graph, temporal) flow through a separate pool against this URL, offloading the primary. Typically points to a read-only endpoint (CNPG's <cluster>-ro service or Aurora reader endpoint).
 # HINDSIGHT_API_MIGRATION_DATABASE_URL=  # Direct PostgreSQL URL for migrations (bypasses PgBouncer). Falls back to DATABASE_URL.
 # HINDSIGHT_API_DATABASE_SCHEMA=public  # PostgreSQL schema name (default: public)
+# HINDSIGHT_API_MIGRATION_CONCURRENCY=1  # Tenant schemas to migrate concurrently (PG only, each in its own process; per-schema work stays sequential). Each worker has ~1-2s startup cost + uses ~3 DB connections, so it only pays off with many schemas (tens+) or slow migrations; keep concurrency*3 <= spare max_connections. Default: 1 (sequential).
 
 # Vector Extension (Optional - uses pgvector by default)
 # Options: "pgvector" (default), "vchord", "pgvectorscale" (DiskANN)
@@ -79,6 +87,18 @@ HINDSIGHT_API_LOG_LEVEL=info
 # korean_lindera/lindera(korean), ngram(min,max), edge_ngram(min,max)
 # HINDSIGHT_API_TEXT_SEARCH_EXTENSION_PG_SEARCH_TOKENIZER=
 
+# File Parser (Optional - uses markitdown by default)
+# HINDSIGHT_API_FILE_PARSER=markitdown
+# Enable image OCR for MarkItDown using an OpenAI-compatible OCR/vision endpoint.
+# These OCR settings are independent from HINDSIGHT_API_LLM_* because MarkItDown
+# uses the OpenAI SDK directly and requires Chat Completions image input support.
+# When OCR is enabled, API_KEY, BASE_URL, and MODEL are required.
+# HINDSIGHT_API_FILE_PARSER_MARKITDOWN_OCR_ENABLED=false
+# HINDSIGHT_API_FILE_PARSER_MARKITDOWN_OCR_API_KEY=
+# HINDSIGHT_API_FILE_PARSER_MARKITDOWN_OCR_BASE_URL=
+# HINDSIGHT_API_FILE_PARSER_MARKITDOWN_OCR_MODEL=
+# HINDSIGHT_API_FILE_PARSER_MARKITDOWN_OCR_PROMPT=
+
 # Embeddings Configuration (Optional - uses local by default)
 # Provider: "local" (default), "onnx", "tei", "openai", "cohere", "google", "openrouter", "zeroentropy", "litellm", or "litellm-sdk"
 # HINDSIGHT_API_EMBEDDINGS_PROVIDER=local
@@ -142,6 +162,10 @@ HINDSIGHT_API_LOG_LEVEL=info
 # Custom service name and environment (optional, defaults: hindsight-api, development)
 # HINDSIGHT_API_OTEL_SERVICE_NAME=hindsight-production
 # HINDSIGHT_API_OTEL_DEPLOYMENT_ENVIRONMENT=production
+#
+# Expose async-operation queue + consolidation-backlog gauges on /metrics.
+# Runs periodic per-schema COUNT queries on a background task (disabled by default).
+# HINDSIGHT_API_METRICS_BACKLOG_ENABLED=true
 
 # -----------------------------------------------------------------------------
 # Control Plane (Optional)

.github/workflows/deploy-docs.yml

@@ -22,6 +22,8 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v6
+        with:
+          fetch-depth: 0 # fetch tags so check-released-integrations can see them
       - uses: actions/setup-node@v6
         with:
           node-version: 20

.github/workflows/perf-test.yml

@@ -23,6 +23,7 @@ on:
           - retain
           - recall
           - recall-with-observations
+          - recall-temporal
           - consolidation
           - graph-maintenance
         default: ""

.github/workflows/release-integration.yml

@@ -9,7 +9,11 @@ jobs:
   publish:
     runs-on: ubuntu-latest
     permissions:
-      id-token: write  # for PyPI trusted publishing
+      id-token: write  # for PyPI trusted publishing + build-provenance attestations
+      attestations: write  # for actions/attest-build-provenance (Obsidian assets)
+      # No `contents: write`: we never create releases in this repo. The Obsidian
+      # plugin's distribution release is pushed to its dedicated repo using
+      # OBSIDIAN_DIST_TOKEN (see the "Mirror Obsidian plugin" step below).
 
     steps:
     - uses: actions/checkout@v6
@@ -112,6 +116,71 @@ jobs:
       working-directory: ./hindsight-integrations/${{ steps.info.outputs.integration }}
       run: npm run build
 
+    # Build-provenance attestations for the Obsidian release assets (community-store
+    # recommendation). Runs after the build so main.js exists. The assets are
+    # released in the dedicated repo while the build runs here, so users verify at
+    # owner scope: `gh attestation verify main.js --owner vectorize-io`.
+    - name: Attest Obsidian plugin build provenance
+      if: steps.type.outputs.type == 'typescript' && steps.info.outputs.integration == 'obsidian'
+      uses: actions/attest-build-provenance@v2
+      with:
+        subject-path: |
+          hindsight-integrations/obsidian/main.js
+          hindsight-integrations/obsidian/styles.css
+
+    # ── Obsidian plugin — mirror to its dedicated repo + cut the BRAT release ──
+    # We do NOT create a GitHub Release in this monorepo: per-integration
+    # releases pollute the repo's release list (it's for the core product) and
+    # steal the "Latest" badge, and BRAT / the community store read a repo's
+    # *latest* release — not a tag — so they can't target a tag in a monorepo.
+    #
+    # Instead this monorepo stays the source of truth, and on each obsidian
+    # release we mirror hindsight-integrations/obsidian/ → the *root* of
+    # github.com/vectorize-io/hindsight-obsidian (git subtree, history
+    # preserved) and cut the BRAT / community-store release *there*.
+    #
+    # Requires secret OBSIDIAN_DIST_TOKEN — a token with `contents: write` on
+    # vectorize-io/hindsight-obsidian (fine-grained PAT or app installation
+    # token). The dedicated repo is generated; never edit it directly.
+    - name: Mirror Obsidian plugin to its dedicated repo
+      if: steps.type.outputs.type == 'typescript' && steps.info.outputs.integration == 'obsidian'
+      env:
+        DIST_TOKEN: ${{ secrets.OBSIDIAN_DIST_TOKEN }}
+      run: |
+        set -euo pipefail
+        VERSION="${{ steps.info.outputs.version }}"
+        DIST_REPO="vectorize-io/hindsight-obsidian"
+        OBS_DIR="hindsight-integrations/obsidian"
+
+        # `git subtree split` needs full history; the default checkout is shallow.
+        git fetch --unshallow 2>/dev/null || true
+        git config user.name "github-actions[bot]"
+        git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
+
+        # The runner injects the default GITHUB_TOKEN as an http.extraheader via
+        # an *included* config file (/home/runner/work/_temp/git-credentials-*.config),
+        # so `git config --local --unset-all` can't remove it and it authenticates
+        # the push as github-actions[bot] (no access to the dedicated repo → 403).
+        # The documented way to drop an inherited extraheader is to RESET the list
+        # with an empty value: since command-line `-c` is read last, the empty
+        # value clears the accumulated headers (including the included one) at
+        # request-build time. The dist token then comes from the push URL → a
+        # single Authorization header.
+        git subtree split --prefix="$OBS_DIR" -b _obs_dist
+        git -c "http.https://github.com/.extraheader=" \
+          push "https://x-access-token:${DIST_TOKEN}@github.com/${DIST_REPO}.git" _obs_dist:main
+
+        # Cut the BRAT / community-store release. Bare version tag (e.g. 0.1.0)
+        # to match manifest.json — idempotent so re-runs just refresh the assets.
+        export GH_TOKEN="$DIST_TOKEN"
+        ASSETS="$OBS_DIR/main.js $OBS_DIR/manifest.json $OBS_DIR/styles.css"
+        NOTES="Hindsight for Obsidian v${VERSION}. Install via BRAT (add ${DIST_REPO}) or copy main.js/manifest.json/styles.css into <vault>/.obsidian/plugins/hindsight/."
+        if gh release view "$VERSION" --repo "$DIST_REPO" >/dev/null 2>&1; then
+          gh release upload "$VERSION" $ASSETS --repo "$DIST_REPO" --clobber
+        else
+          gh release create "$VERSION" $ASSETS --repo "$DIST_REPO" --title "$VERSION" --notes "$NOTES"
+        fi
+
     - name: Publish TypeScript package to npm
       if: steps.type.outputs.type == 'typescript'
       working-directory: ./hindsight-integrations/${{ steps.info.outputs.integration }}
@@ -121,7 +190,12 @@ jobs:
         EXIT_CODE=$?
         echo "$OUTPUT"
         if [ $EXIT_CODE -ne 0 ]; then
-          if echo "$OUTPUT" | grep -q "cannot publish over"; then
+          # Treat "already published" as success so re-pointed-tag re-runs stay green.
+          # "cannot publish over" = the version exists. TLOG_CREATE_ENTRY_ERROR / 409
+          # "equivalent entry already exists in the transparency log" = the identical
+          # --provenance artifact was already logged on a prior run (Sigstore tlog is
+          # idempotent); the package is published, so this is benign.
+          if echo "$OUTPUT" | grep -qE "cannot publish over|TLOG_CREATE_ENTRY_ERROR|already exists in the transparency log"; then
             echo "Package version already published, skipping..."
             exit 0
           fi