Skip to content

Add live conversation signposts#497

Open
dtrn2048 wants to merge 7 commits intomainfrom
feat/signposts-main
Open

Add live conversation signposts#497
dtrn2048 wants to merge 7 commits intomainfrom
feat/signposts-main

Conversation

@dtrn2048
Copy link
Copy Markdown
Contributor

@dtrn2048 dtrn2048 commented Mar 26, 2026

Summary

  • add the live per-conversation signposting pipeline, schema, task orchestration, and host-facing UI
  • include the signposts localization follow-up so Portal Editor, conversation overview, and Host Guide render readable copy in the priority locales
  • keep v1 scoped to host-only, per-conversation signposts with project-level configuration

Validation

  • uv run mypy dembrane/signposting.py dembrane/service/conversation.py dembrane/tasks.py dembrane/transcribe.py
  • uv run pytest tests/test_signposting.py tests/service/test_conversation_signposting_service.py tests/test_tasks_webhook.py -q
  • uv run ruff check
  • pnpm --dir echo/frontend messages:extract
  • pnpm --dir echo/frontend messages:compile
  • pnpm --dir echo/frontend build

Deploy notes

  • main will still need the Directus schema sync for the new conversation_signpost collection and related fields
  • no new signposting-specific env vars are required

Summary by CodeRabbit

  • New Features

    • Live conversation signposts surfaced in the UI (Host Guide & Conversation Overview) with category badges, counts, titles, summaries and evidence quotes
    • Project settings to enable/disable live signposts and add focus terms
  • Tests

    • New end-to-end and unit tests covering signpost generation, processing, locking and task scheduling
  • Chores

    • Test environment credentials cleared; added local dev endpoints and new signpost translation entries across locales

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Mar 26, 2026

Walkthrough

Implements live conversation signposts: Directus schema additions, new backend signposting logic and tasks (LLM-driven create/update/resolve), Redis coordination locks and scheduler job, transcription integration to mark chunks ready, frontend types/hooks/components/locales, Cypress E2E test, and unit tests for service/task behavior.

Changes

Cohort / File(s) Summary
Directus schema & relations
echo/directus/sync/snapshot/collections/conversation_signpost.json, echo/directus/sync/snapshot/fields/conversation_signpost/*, echo/directus/sync/snapshot/fields/conversation/signposts.json, echo/directus/sync/snapshot/fields/conversation_chunk/signpost_*.json, echo/directus/sync/snapshot/fields/project/*.json, echo/directus/sync/snapshot/relations/conversation_signpost/*
Adds conversation_signpost collection, multiple field snapshots (title, summary, category, confidence, ids, timestamps), project flags and focus-terms, conversation alias field, and FK relations to conversation/conversation_chunk.
Backend signposting implementation
echo/server/dembrane/signposting.py, echo/server/prompt_templates/live_conversation_signposts.en.jinja
New module that builds LLM prompt, validates JSON response, generates create/update/resolve operations, and applies them to Directus; adds strict JSON schema and Jinja prompt template.
Backend service & tasks
echo/server/dembrane/service/conversation.py, echo/server/dembrane/tasks.py, echo/server/dembrane/coordination.py, echo/server/dembrane/conversation_utils.py, echo/server/dembrane/scheduler.py, echo/server/dembrane/transcribe.py
Adds ConversationService signposting helpers (mark/list/create/update), task actors to refresh and catch-up signposts, per-conversation signposting locks (atomic SET nx+ex), utility to collect pending conversations, scheduler job (every 2 minutes), and transcription hooks to mark chunks signpost-ready and enqueue refreshes.
Server helpers & fixes
echo/server/dembrane/conversation_utils.py, echo/server/dembrane/coordination.py
Introduces helper to collect conversations needing signposts; refactors Redis lock acquisition to atomic SET nx+ex and adds signposting lock helpers/cleanup.
Frontend types, hooks & components
echo/frontend/src/lib/typesDirectus.d.ts, echo/frontend/src/hooks/useLiveConversationSignposts.ts, echo/frontend/src/components/conversation/ConversationSignpostsSection.tsx, echo/frontend/src/components/conversation/hooks/index.ts, echo/frontend/src/components/project/ProjectPortalEditor.tsx, echo/frontend/src/routes/project/HostGuidePage.tsx, echo/frontend/src/routes/project/conversation/ProjectConversationOverview.tsx, echo/frontend/src/routes/project/ProjectRoutes.tsx
Adds ConversationSignpost TS types, polling hook for live signposts, UI section/component to render signposts, portal editor fields (toggle + focus terms), host guide live panel, and integrates signpost fields into data queries and rendering when enabled.
Frontend localization
echo/frontend/src/locales/{de-DE,en-US,es-ES,fr-FR,it-IT,nl-NL}.po
Adds/updates translation entries and source references for signposts UI strings, categories, timestamps, explanatory/help text, and focus-terms across locales.
Frontend tests & E2E config
echo/cypress/cypress.env.json, echo/cypress/e2e/suites/34-live-signposts.cy.js, echo/cypress/cypress.config.js, echo/cypress/support/functions/login/index.js
Clears hardcoded creds in cypress.env.json, adds local envs, changes Cypress env precedence to favor process vars, validates both email/password on login, and adds E2E test that enables signposting, seeds Directus signpost data, and asserts UI propagation.
Backend unit tests & test updates
echo/server/tests/service/test_conversation_signposting_service.py, echo/server/tests/test_signposting.py, echo/server/tests/test_tasks_webhook.py
Adds tests for ConversationService create-chunk behavior (marks signpost-ready and enqueues), signposting apply/refresh logic, task locking/requeueing, catch-up behavior, and updates transcript-correction tests to assert mark_signpost_ready.
Misc: scheduler & prompt template
echo/server/prompt_templates/live_conversation_signposts.en.jinja, echo/server/dembrane/scheduler.py
Adds Jinja LLM prompt template and a CronTrigger job to run catch-up every 2 minutes.
Minor: Cypress env parsing
echo/cypress/cypress.env.json, echo/cypress/cypress.config.js, echo/cypress/support/functions/login/index.js
Replaced embedded credentials with empty strings, added local endpoints, and made Cypress config derive auth from process env with fallbacks; updated login validation and error messaging.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~75 minutes

Possibly related PRs

LGTM.

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'Add live conversation signposts' directly describes the main feature being introduced—a live per-conversation signposting pipeline with UI and task orchestration across frontend, backend, and schema layers.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/signposts-main

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 23

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@echo/cypress/cypress.env.json`:
- Around line 40-47: The cypress.env.json currently contains plaintext
credentials under the "auth" object ("email" and "password"); remove these
values from the committed file and replace them with references to environment
variables or a local untracked override. Specifically, delete or empty the
"auth" -> "email" and "auth" -> "password" entries in cypress.env.json and
update test/CI usage to read from CYPRESS_EMAIL and CYPRESS_PASSWORD (or a local
cypress.env.local that's .gitignored) so credentials are provided via CI secrets
or untracked local config instead of being committed.

In `@echo/cypress/e2e/suites/34-live-signposts.cy.js`:
- Around line 124-126: The test body for it("shows seeded signposts in portal
settings, conversation overview, and host guide") declares projectId/locale and
performs teardown calls deleteProject and logout inside the test, which can be
skipped on failure; move the cleanup into an afterEach hook (or a dedicated
finally-style helper) so deleteProject(projectId) and logout() always run
regardless of test outcome; update the spec to set projectId in the test but
perform deleteProject and logout in afterEach (or call the helper) and reference
the same projectId variable used by the it block to ensure proper cleanup.
- Around line 139-143: Replace the brittle cy.wait(3000) after typing into
getFocusTermsTextarea() with a deterministic sync point: either intercept the
persistence network request with cy.intercept(...) and wait on its alias after
the .type(...) call, or assert the DOM saved-state signal (e.g., a "Saved" badge
or disabled spinner) becomes visible/contains expected text; update the test
around getFocusTermsTextarea() so it waits for that request alias or the
saved-state element instead of using a fixed delay.

In
`@echo/directus/sync/snapshot/fields/conversation_chunk/signpost_processed_at.json`:
- Around line 12-18: The signpost_processed_at field is currently editable in
the UI; update the field metadata in signpost_processed_at.json to lock it down
by setting "hidden": true and "readonly": true (and consider setting
"searchable": false) so the worker-owned timestamp cannot be manually edited or
surfaced in Directus UI; keep the "interface": "datetime" but ensure the
"hidden" and "readonly" flags are changed on the signpost_processed_at field.

In
`@echo/directus/sync/snapshot/fields/conversation_chunk/signpost_ready_at.json`:
- Around line 34-37: Add a DB index on the signpost_ready_at field to speed
worker polling queries: update the schema for the signpost_ready_at column (the
JSON entry "signpost_ready_at") to set is_indexed: true and add a matching index
definition; if your workers commonly filter by both readiness and a processed
marker, create a composite index that includes signpost_ready_at plus the
processed flag/timestamp field (e.g. processed_at or is_processed) so queries
filtering by readiness and processed state use the index.

In `@echo/directus/sync/snapshot/fields/conversation_signpost/category.json`:
- Around line 36-37: The category field in conversation_signpost currently
allows null/absent values; change its schema to make category non-nullable and
required by setting "required": true and "is_nullable": false for the "category"
field (update the entries where "required": false and "is_nullable": true), so
records cannot be created without a category and downstream rendering/filtering
remains deterministic.

In
`@echo/directus/sync/snapshot/fields/conversation_signpost/conversation_id.json`:
- Around line 20-21: Make the conversation_id field non-nullable/mandatory:
change its "required" from false to true and ensure "nullable" is set to false
(or the equivalent non-null flag) in conversation_signpost's JSON definition
(the conversation_id field) and apply the same change to the other occurrence
referenced (lines ~39-47) so signposts cannot be persisted without a
conversation FK; mirror the pattern used by conversation_chunk to enforce
per-conversation integrity.

In
`@echo/directus/sync/snapshot/fields/conversation_signpost/evidence_chunk_id.json`:
- Line 41: The foreign-key field "evidence_chunk_id" currently has "is_indexed":
false; change it to "is_indexed": true in the
conversation_signpost/evidence_chunk_id JSON so the schema creates an index for
queries joining/filtering by this FK, and if your deployment requires explicit
migrations also add a DB migration to create an index on
conversation_signpost.evidence_chunk_id so production will get the index.

In `@echo/directus/sync/snapshot/fields/conversation_signpost/status.json`:
- Around line 28-29: The status field currently allows null/missing values (it
has "required": false and "is_nullable": true), which lets records omit status
and breaks deterministic signpost state; update the status field JSON to make
the field required and non-nullable by changing "required" to true and
"is_nullable" to false so every record has a defined status value (referencing
the "status" field and its "required" / "is_nullable" keys in the schema).

In `@echo/directus/sync/snapshot/fields/conversation_signpost/updated_at.json`:
- Around line 16-22: The updated_at field is marked editable but uses the
server-managed special key "date-updated"; change the JSON for the updated_at
field to set "readonly": true to prevent client-side edits and preserve audit
integrity, ensuring the "special": ["date-updated"] entry remains present and
unchanged so the server continues to manage timestamps.

In `@echo/frontend/src/components/conversation/hooks/index.ts`:
- Around line 885-898: The signpost selection omits conversation_id causing a
type-contract mismatch when the results are cast to ConversationSignpost[];
update the signposts selection object (the array that currently lists
"id","category","title","summary","evidence_quote","status","confidence","created_at","updated_at","evidence_chunk_id")
to include "conversation_id" so the shape returned by the query matches the
ConversationSignpost type used downstream.

In `@echo/frontend/src/lib/typesDirectus.d.ts`:
- Around line 293-296: The Project type's language property is too narrow and
doesn't include the newly supported languages, causing saved values
("de","fr","es","it") to be rejected; update the language union on the Project
(the language property in echo/frontend/src/lib/typesDirectus.d.ts) to include
"de", "fr", "es", and "it" in addition to "en", "nl", "multi" and null so the
type matches ProjectPortalEditor persistence.

In `@echo/frontend/src/locales/it-IT.po`:
- Around line 1989-1991: Update the user-facing copy in ProjectPortalEditor.tsx
to use "language model" instead of "AI" for the feature description (the string
currently matching the msgid shown in the .po file), then regenerate the
localization catalog so echo/frontend/src/locales/it-IT.po reflects that change
(also fix the other instance referenced around the second set of msgid/msgstr).
Ensure the updated source string in ProjectPortalEditor.tsx exactly replaces
"AI" with "language model" and run the i18n extraction/compile step to update
all .po/.json entries.
- Around line 463-467: The Italian locale file
echo/frontend/src/locales/it-IT.po contains blank msgstr entries (e.g., msgid
"{0} live" associated with ConversationSignpostsSection.tsx) so the signposts UI
will appear in English; update the it-IT translations for these msgid keys
(including the other reported missing entries referenced in the review) by
supplying proper Italian msgstr values, then re-run the lingui/cli extract and
compile steps to regenerate the .po/.json artifacts for frontend/src/locales so
all supported languages (en-US, nl-NL, de-DE, fr-FR, es-ES, it-IT) are fully
populated.

In `@echo/frontend/src/locales/nl-NL.po`:
- Around line 5361-5364: The translation for msgid "Verify Topics" is incorrect
Dutch ("Verifiëren Themen"); update the msgstr in nl-NL.po to a proper Dutch
phrase (e.g., "Onderwerpen verifiëren") so it contains only Dutch; locate the
entry in echo/frontend/src/locales/nl-NL.po (the block referencing
src/components/project/ProjectPortalEditor.tsx:1056) and replace the msgstr
value accordingly, then run the translation pipeline/CI checks to ensure no
remaining mixed-language entries.
- Around line 1839-1842: The source string in ProjectPortalEditor.tsx currently
uses “AI-powered responses”; update that UI copy to use “language model” (e.g.,
change the phrase to “language model-powered responses” or “language model
responses” in the JSX where the msgid originates), then re-run the i18n
extraction/regeneration process so the .po files (including nl-NL.po) are
updated accordingly; look for the string in ProjectPortalEditor.tsx (the
JSX/label that produces the msgid shown) and update it, then run your usual
localization tooling to regenerate messages.
- Around line 1553-1555: The Dutch translation uses formal pronouns ("u/uw/U")
but our style guide requires informal "je/jij"; update the source strings in
ProjectPortalEditor.tsx (the msgid that starts "Determines under which GDPR
legal basis personal data is processed..." and the other source strings
referenced around the msgids at the locations noted) to replace "u/uw/U" with
"je/jij" in the original English/source text, then re-extract the .po entries so
the nl-NL.po (lines around the shown msgid and the other affected msgids) are
regenerated and can be translated consistently using "je/jij".
- Around line 4855-4858: The Dutch translation is wrong because the source
prompt string in ProjectPortalEditor is being truncated/duplicated during
extraction; update the original English source string in
src/components/project/ProjectPortalEditor.tsx (the msgid: "This prompt guides
how the AI responds to participants. Customize it to shape the type of feedback
or engagement you want to encourage.") so it is exact and not split across
concatenated literals, then re-run the localization extraction/compile pipeline
to regenerate the .po catalogs (do not hand-edit nl-NL.po); ensure the
extraction step picks up the full string and that any i18n helper functions
around ProjectPortalEditor.tsx preserve the complete sentence.

In `@echo/server/dembrane/conversation_utils.py`:
- Around line 182-205: The directus.get_items call in conversation_utils.py is
blocking and must be moved off the event loop; wrap the call with
run_in_thread_pool from dembrane.async_helpers (or make the containing function
async and await run_in_thread_pool), e.g. replace the directus.get_items
invocation with await run_in_thread_pool(directus.get_items, "conversation",
{...}) and add the run_in_thread_pool import; ensure the surrounding function
signature is async (or already async) and propagate awaits accordingly so the
blocking IO is executed in a thread pool.
- Around line 209-214: The collector loop that appends to conversation_ids can
crash in the except handler because it accesses conversation['id'] again; change
the error logging in the loop (the for conversation in response block that
appends to conversation_ids and calls logger.error) to avoid re-accessing the
missing key by using a safe accessor or representation (e.g.,
conversation.get("id") or repr(conversation)) and include the caught exception
details, and ensure the loop continues (do not raise) so a missing id doesn't
break collection.

In `@echo/server/dembrane/coordination.py`:
- Around line 454-457: Replace the non-atomic setnx + expire pattern at all
seven lock sites (processing started lock, finish lock, finalize lock, chunk
decrement mark, summarize lock, signposting lock, AAI webhook processing lock)
with a single atomic Redis SET call using NX and EX options (e.g.
client.set(key, "1", nx=True, ex=_..._LOCK_TTL_SECONDS)); use the returned
truthy value to detect acquisition and perform the same logging/branching as
before (e.g. logger.debug on success), and remove the separate client.expire
calls so there is no TOCTOU window.

In `@echo/server/dembrane/service/conversation.py`:
- Around line 605-623: mark_chunks_signpost_processed currently loops and calls
client.update_item per chunk which may become a bottleneck; change it to use a
single batch update call when the client supports it (e.g., a method like
client.update_items / client.batch_update) by building a list/dict of updates
for "conversation_chunk" with id and {"signpost_processed_at":
processed_timestamp}, call the batch API once, and keep the existing per-item
loop as a fallback if the batch method is absent; reference the function
mark_chunks_signpost_processed, the variable processed_timestamp, and the
client.update_item call so you add the batch path while preserving current
behavior.

In `@echo/server/prompt_templates/live_conversation_signposts.en.jinja`:
- Around line 39-48: Add an explicit JSON schema example to the "Output format"
section showing the exact structure for the top-level keys "create", "update",
and "resolve" (e.g., an example object with "create": [{...}], "update":
[{...}], "resolve": ["..."]) and explicitly require the evidence field
"evidence_chunk_id" on every create and update entry; update the prose near
"Output format" to state that each object in "create" and "update" must include
evidence_chunk_id and show it in the example to improve LLM reliability and
traceability (refer to the "Output format" heading and the keys "create",
"update", "resolve" and the field name "evidence_chunk_id").
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: b1316ea5-5578-4247-b455-21eca86eddc8

📥 Commits

Reviewing files that changed from the base of the PR and between 2c73a4a and cb40548.

📒 Files selected for processing (52)
  • echo/cypress/cypress.env.json
  • echo/cypress/e2e/suites/34-live-signposts.cy.js
  • echo/directus/sync/snapshot/collections/conversation_signpost.json
  • echo/directus/sync/snapshot/fields/conversation/signposts.json
  • echo/directus/sync/snapshot/fields/conversation_chunk/signpost_processed_at.json
  • echo/directus/sync/snapshot/fields/conversation_chunk/signpost_ready_at.json
  • echo/directus/sync/snapshot/fields/conversation_signpost/category.json
  • echo/directus/sync/snapshot/fields/conversation_signpost/confidence.json
  • echo/directus/sync/snapshot/fields/conversation_signpost/conversation_id.json
  • echo/directus/sync/snapshot/fields/conversation_signpost/created_at.json
  • echo/directus/sync/snapshot/fields/conversation_signpost/evidence_chunk_id.json
  • echo/directus/sync/snapshot/fields/conversation_signpost/evidence_quote.json
  • echo/directus/sync/snapshot/fields/conversation_signpost/id.json
  • echo/directus/sync/snapshot/fields/conversation_signpost/status.json
  • echo/directus/sync/snapshot/fields/conversation_signpost/summary.json
  • echo/directus/sync/snapshot/fields/conversation_signpost/title.json
  • echo/directus/sync/snapshot/fields/conversation_signpost/updated_at.json
  • echo/directus/sync/snapshot/fields/project/is_signposting_enabled.json
  • echo/directus/sync/snapshot/fields/project/signposting_focus_terms.json
  • echo/directus/sync/snapshot/relations/conversation_signpost/conversation_id.json
  • echo/directus/sync/snapshot/relations/conversation_signpost/evidence_chunk_id.json
  • echo/frontend/src/components/conversation/ConversationSignpostsSection.tsx
  • echo/frontend/src/components/conversation/hooks/index.ts
  • echo/frontend/src/components/project/ProjectPortalEditor.tsx
  • echo/frontend/src/hooks/useLiveConversationSignposts.ts
  • echo/frontend/src/lib/typesDirectus.d.ts
  • echo/frontend/src/locales/de-DE.po
  • echo/frontend/src/locales/de-DE.ts
  • echo/frontend/src/locales/en-US.po
  • echo/frontend/src/locales/en-US.ts
  • echo/frontend/src/locales/es-ES.po
  • echo/frontend/src/locales/es-ES.ts
  • echo/frontend/src/locales/fr-FR.po
  • echo/frontend/src/locales/fr-FR.ts
  • echo/frontend/src/locales/it-IT.po
  • echo/frontend/src/locales/it-IT.ts
  • echo/frontend/src/locales/nl-NL.po
  • echo/frontend/src/locales/nl-NL.ts
  • echo/frontend/src/routes/project/HostGuidePage.tsx
  • echo/frontend/src/routes/project/ProjectRoutes.tsx
  • echo/frontend/src/routes/project/conversation/ProjectConversationOverview.tsx
  • echo/server/dembrane/conversation_utils.py
  • echo/server/dembrane/coordination.py
  • echo/server/dembrane/scheduler.py
  • echo/server/dembrane/service/conversation.py
  • echo/server/dembrane/signposting.py
  • echo/server/dembrane/tasks.py
  • echo/server/dembrane/transcribe.py
  • echo/server/prompt_templates/live_conversation_signposts.en.jinja
  • echo/server/tests/service/test_conversation_signposting_service.py
  • echo/server/tests/test_signposting.py
  • echo/server/tests/test_tasks_webhook.py

Harden the new signposting coordination and test paths so the validated review findings no longer block merge.

Made-with: Cursor
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
echo/cypress/cypress.env.json (1)

16-47: ⚠️ Potential issue | 🟠 Major

Add directusUrl to the non-local env blocks too.

The new live-signposts spec seeds Directus via Cypress.env("directusUrl"). Right now only local defines it, so staging / testing / prod runs will silently fall back to http://localhost:8055 and point seeding at the wrong backend.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@echo/cypress/cypress.env.json` around lines 16 - 47, Add the missing
"directusUrl" key to the "staging", "prod", and "testing" environment objects in
cypress.env.json so Cypress.env("directusUrl") points to the correct Directus
instance instead of defaulting to localhost; update the "staging", "prod", and
"testing" blocks (the JSON objects named "staging", "prod", "testing") to
include a "directusUrl" property with the appropriate environment-specific URL
values.
♻️ Duplicate comments (1)
echo/server/dembrane/conversation_utils.py (1)

182-205: ⚠️ Potential issue | 🟠 Major

Move Directus fetch off the blocking path.

directus.get_items(...) is still invoked directly in backend code. This should be moved to run_in_thread_pool and awaited to avoid blocking execution paths.

⚙️ Proposed fix
+from dembrane.async_helpers import run_in_thread_pool
 ...
-def collect_conversations_needing_signposts(limit: int = 50) -> List[str]:
+async def collect_conversations_needing_signposts(limit: int = 50) -> List[str]:
 ...
-    response = directus.get_items(
-        "conversation",
-        {
-            "query": {
-                "filter": {
-                    "project_id": {
-                        "is_signposting_enabled": {"_eq": True},
-                    },
-                    "chunks": {
-                        "_some": {
-                            "_and": [
-                                {"signpost_ready_at": {"_nnull": True}},
-                                {"signpost_processed_at": {"_null": True}},
-                                {"transcript": {"_nempty": True}},
-                            ]
-                        }
-                    },
-                },
-                "fields": ["id"],
-                "sort": ["created_at"],
-                "limit": limit,
-            },
-        },
-    )
+    response = await run_in_thread_pool(
+        directus.get_items,
+        "conversation",
+        {
+            "query": {
+                "filter": {
+                    "project_id": {
+                        "is_signposting_enabled": {"_eq": True},
+                    },
+                    "chunks": {
+                        "_some": {
+                            "_and": [
+                                {"signpost_ready_at": {"_nnull": True}},
+                                {"signpost_processed_at": {"_null": True}},
+                                {"transcript": {"_nempty": True}},
+                            ]
+                        }
+                    },
+                },
+                "fields": ["id"],
+                "sort": ["created_at"],
+                "limit": limit,
+            },
+        },
+    )

As per coding guidelines "Always wrap blocking I/O calls using run_in_thread_pool from dembrane.async_helpers in backend code. Wrap calls to directus.* ... Prefer converting endpoints to async def and await results rather than using synchronous functions".

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@echo/server/dembrane/conversation_utils.py` around lines 182 - 205, The
directus.get_items(...) call is a blocking I/O call and must be moved off the
event loop: import and use run_in_thread_pool from dembrane.async_helpers, call
run_in_thread_pool(lambda: directus.get_items(...)) and await its result instead
of invoking directus.get_items synchronously; if the surrounding function (e.g.,
the function containing the snippet in conversation_utils.py) is not async,
convert it to async def and await the run_in_thread_pool call, preserving the
same query dict and returning the response as before.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Outside diff comments:
In `@echo/cypress/cypress.env.json`:
- Around line 16-47: Add the missing "directusUrl" key to the "staging", "prod",
and "testing" environment objects in cypress.env.json so
Cypress.env("directusUrl") points to the correct Directus instance instead of
defaulting to localhost; update the "staging", "prod", and "testing" blocks (the
JSON objects named "staging", "prod", "testing") to include a "directusUrl"
property with the appropriate environment-specific URL values.

---

Duplicate comments:
In `@echo/server/dembrane/conversation_utils.py`:
- Around line 182-205: The directus.get_items(...) call is a blocking I/O call
and must be moved off the event loop: import and use run_in_thread_pool from
dembrane.async_helpers, call run_in_thread_pool(lambda: directus.get_items(...))
and await its result instead of invoking directus.get_items synchronously; if
the surrounding function (e.g., the function containing the snippet in
conversation_utils.py) is not async, convert it to async def and await the
run_in_thread_pool call, preserving the same query dict and returning the
response as before.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 2638370e-7a11-4464-aae6-9794bdc55373

📥 Commits

Reviewing files that changed from the base of the PR and between cb40548 and c6b81fb.

📒 Files selected for processing (8)
  • echo/cypress/cypress.config.js
  • echo/cypress/cypress.env.json
  • echo/cypress/e2e/suites/34-live-signposts.cy.js
  • echo/cypress/support/functions/login/index.js
  • echo/directus/sync/snapshot/fields/conversation_chunk/signpost_processed_at.json
  • echo/frontend/src/components/conversation/hooks/index.ts
  • echo/server/dembrane/conversation_utils.py
  • echo/server/dembrane/coordination.py

Apply the ruff import and dead-code cleanups from the backend lint pass so the branch stays green.

Made-with: Cursor
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
echo/server/dembrane/tasks.py (1)

351-355: ⚠️ Potential issue | 🟠 Major

Redact fallback transcripts before marking them signpost-ready.

Line 355 now feeds the fallback transcript into live signposting even when use_pii_redaction=True. In the non-anonymized redaction mode, fallback_transcript is still raw here, so a correction failure can persist PII into signpost evidence despite redaction being requested.

Patch
         try:
+            fallback_to_save = fallback_transcript or "[Nothing to transcribe]"
+            if use_pii_redaction and not anonymize_transcripts:
+                from dembrane.pii_regex import regex_redact_pii
+
+                fallback_to_save = (
+                    regex_redact_pii(fallback_to_save) or "[Nothing to transcribe]"
+                )
+
             _save_transcript(
                 chunk_id,
-                fallback_transcript or "[Nothing to transcribe]",
+                fallback_to_save,
                 diarization=fallback_diarization,
                 mark_signpost_ready=True,
             )
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@echo/server/dembrane/tasks.py` around lines 351 - 355, The fallback
transcript is being passed raw into _save_transcript (with
mark_signpost_ready=True) which can leak PII when use_pii_redaction is enabled;
before calling _save_transcript use the same redaction path as the normal
transcript flow to produce a redacted_fallback (or set fallback_transcript to
the redacted value) and pass that into _save_transcript instead of the raw
fallback_transcript, ensuring you respect the use_pii_redaction flag and
preserve diarization and mark_signpost_ready parameters.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@echo/server/dembrane/signposting.py`:
- Around line 168-176: The code is hardcoding the prompt language to "en" when
calling render_prompt; update the call to use the project's locale from the
signposting context (e.g., the language value returned by
ConversationService.get_signposting_context, such as project_language or
project_context.language) and keep a safe fallback to "en" if that language is
missing/empty so non-English projects use their localized prompt path; modify
the render_prompt invocation in signposting.py (the call to
render_prompt("live_conversation_signposts", ...)) to pass the project language
variable instead of the literal "en".

---

Outside diff comments:
In `@echo/server/dembrane/tasks.py`:
- Around line 351-355: The fallback transcript is being passed raw into
_save_transcript (with mark_signpost_ready=True) which can leak PII when
use_pii_redaction is enabled; before calling _save_transcript use the same
redaction path as the normal transcript flow to produce a redacted_fallback (or
set fallback_transcript to the redacted value) and pass that into
_save_transcript instead of the raw fallback_transcript, ensuring you respect
the use_pii_redaction flag and preserve diarization and mark_signpost_ready
parameters.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 48fa4126-278c-4a30-9c83-266144c45c6c

📥 Commits

Reviewing files that changed from the base of the PR and between c6b81fb and da35b29.

📒 Files selected for processing (5)
  • echo/server/dembrane/signposting.py
  • echo/server/dembrane/tasks.py
  • echo/server/tests/service/test_conversation_signposting_service.py
  • echo/server/tests/test_signposting.py
  • echo/server/tests/test_tasks_webhook.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant