feat(ingestion): Zep document_id grouping for zep-ingest 0.4.0 - #608
feat(ingestion): Zep document_id grouping for zep-ingest 0.4.0#608jackaldenryan wants to merge 2 commits into
Conversation
…v0.4.0) Group related episodes so extraction can resolve pronouns across chunks and Slack threads, and add a non-blocking post-merge/release smoke suite. Co-authored-by: Cursor <cursoragent@cursor.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| assert [e.data for e in channel_eps] == [ | ||
| next(e.data for e in eps if "Hello world" in e.data), | ||
| next(e.data for e in eps if "Should we deprioritize PROTOTYPE-202?" in e.data), | ||
| next(e.data for e in eps if "https://example.com" in e.data), |
|
|
||
| def test_markup_normalization(self): | ||
| markup = general(load())[2].data | ||
| markup = next(e.data for e in general(load()) if "https://example.com" in e.data) |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
-
Critical —
ingestion/src/zep_ingest/result.py,record_sequential_episode()hunk:created_atvalues are compared lexicographically. Valid RFC3339 timestamps with differing offsets or fractional-second representations do not sort chronologically this way.wait()can therefore poll an earlier episode and return before the actual document tail is processed. Parse timestamps into normalized datetimes before selecting the maximum, and add offset/fractional-format tests. -
Critical —
ingestion/src/zep_ingest/documents.py,document_id_for_path(): short absolute paths are sent verbatim asdocument_id. This exposes local directory/user/customer names to Zep and makes document identity change when the same file moves or runs on another machine, fragmenting re-ingestion. Always hash the path—or derive an explicitly scoped relative/content-based identifier—regardless of length. -
Warning —
.github/workflows/smoke-ingestion.yml, checkout/install steps: theworkflow_runpath checks out current default-branch HEAD anduv syncinstalls the checkout, so the advertised post-release smoke test does not test the release commit or published artifact. Check outgithub.event.workflow_run.head_shaand install the released distribution/version before running the smoke suite.
Email Date headers mix timezone offsets; lexicographic created_at order could poll the wrong saga. Post-release smoke now checks out the release SHA. Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Warning — ingestion/src/zep_ingest/documents.py, document_id_for_path() hunk
document_id is derived from path.resolve(), making the same source file produce different IDs when checked out, mounted, or extracted under another directory. Re-ingesting it from CI, another machine, or a moved workspace creates a separate document instead of updating/grouping the existing one.
Use a location-independent identifier, such as a caller-supplied source-relative path or a stable content/source identifier.
2 participants
Summary
document_idto text files, transcripts, and emails (including single-episode sources) so related chunks share one Zep document.document_id). JSON records stay standalone. The Slackgroupingoption is removed; Zepthread_idis not used.graph.addsendsdocument_idvia a thin wrapper until the SDK types it;wait()polls the latest-created_attail per document saga.ingestion/scripts/smoke.py) runs after merge and after release without blocking either. Version is 0.4.0.Test plan
uv run pytest tests/ -m "not integration and not smoke"(687 passed)slack-pronoun-646362a9); Alice facts resolved for both Shesmainto publishzep-ingest-v0.4.0(do not wait on post-merge CI first)Smoke Ingestion Packageworkflow is wired (needsZEP_API_KEYin theintegrationenvironment)Made with Cursor