[Silthus]

Part of the code-first rebuild of the email send stack; new in the rebuild, with no closed PR to replace: a template send delivers the content Django graded (decision D3 on map #461).

Problem

• A template send on send/email/ right after an upload can deliver the previous content, while Django answered that the caller's template.hash is current.
• Django grades the hash against Postgres. The email goes out from the workflow copy that cdp-api and the hogflow worker cached. A lost or late reload leaves that copy behind for up to two minutes.
• This is the cdp-api and worker half. The Django half, which sends the version, is feat(messaging): send the workflow version a template send by key was checked against #585.

Impact

Pays into Grow Workflows through billed workflows_emails actions per paying org: a developer who uploads a template and sends it at once gets the new content, so the first send from code is right. In scope: CDP email sends.

Merge Danger

Warning

🔴 Merge risk: high · two-way door: a revert sends from the cached copy again, as today

• A version check on the CDP email delivery path changes which jobs run, where a wrong refusal drops an email in silence, so the grade is high even behind the flag.
• Shared hot path: the hogflow worker loads flows for every workflow run. The new branch runs only when state.send.hogFlowVersion is set, which only Django from feat(messaging): send the workflow version a template send by key was checked against #585 sets, behind workflows-email-from-code.
• Every mixed window behaves as today. An older cdp-api strips the unknown field. An older worker ignores it. An older Django doesn't send it. The cross-version runs below show all three.
• Land after feat(cdp): key a code send on the sender the caller chose #580. No order against feat(messaging): send the workflow version a template send by key was checked against #585 is needed. The fix works once feat(messaging): send the workflow version a template send by key was checked against #585 and this PR's workers are both live.

Changes

Kind	Files	Lines
Production	5	+101 −12
Tests	3	+214 −0

• A template send uses a workflow version at least as new as the one Django graded. The accepted send can carry hogFlowVersion. When the cached copy is older, cdp-api and the hogflow worker read the workflow from the primary.
• One primary read per workflow serves a burst. Concurrent sends after one upload share one read, and its fresh copy goes into the cache. An older replica load cannot replace it.
• If even the primary is older, nothing is sent. cdp-api queues nothing and answers Django with an internal 503. Django answers the caller with the contract's 503 delivery_unavailable, and puts the version lag in detail and why (feat(messaging): send the workflow version a template send by key was checked against #585).
• The reason stays internal. workflow_version_behind appears only in the body cdp-api sends to Django, in the metric label and in the log. No public answer carries it.
• A queued job in that state fails in the worker, with a log line that names both versions and a workflow_version_behind count. Its email status records the send as failed, as for a workflow that is gone. Versions only go up, so this state should not happen.
• A job whose workflow is gone counts too. The not-found branch now adds to the same failed-jobs counter with reason: 'workflow_unavailable', so the counter covers every job the worker fails before running it.
• feat(cdp): deliver a code send's content on its own transactional email route #565's shared read for a missing workflow and this PR's version floor share one read path. A send without a version takes a read it joined as it is; a send with a version reads again only when the joined copy is older.
• The version stays out of the idempotency fingerprint, so a keyed retry that Django grades after an upload still replays.
• Why a version floor and not a hash key: Node has no copy of the content hash. The worker, not cdp-api, decides which content goes out. The template save writes the content and the bumped workflow version in one transaction. So "at least the graded version" means "the graded content or newer", and both sides already have version. An upload that lands between the grade and the send delivers the newer content. It is never older.
• Limit: the floor covers versioned writes. The asset-URL rewrite command, as fix(messaging): sync the transactional workflow on asset url rewrites #571 reworks it, rewrites a transactional workflow's content before its template sync runs, so the workflow version doesn't bump.
• Running the template pass first bumps the version. But then the real run reports fewer changed rows than the dry run, and its output doesn't say why. So the rewrite stays a known exception, and a send right after one can still deliver the cached copy.
• Rebased onto the reworked feat(cdp): key a code send on the sender the caller chose #580 and master 43333f8. It now builds on feat(cdp): deliver a code send's content on its own transactional email route #565 and feat(cdp): key a code send on the sender the caller chose #580 only. Master's fix(cdp): answer transactional email sends without waiting on kafka PostHog/posthog#125444 (answer before the Kafka flush) and fix(workflows): drop suppressed cc/bcc instead of skipping the email PostHog/posthog#118456 (email status in the worker) are under it.

How did you test this code?

Test seams: cdp-api's accepted-send route over HTTP (cdp-api.serial.test.ts), processInvocations on the hogflow worker, and the hogflow manager service for the read count.

Test rationale, each run red first:

• A send whose graded version is newer than the cached copy that delivers the cached content, in cdp-api and in the worker.
• A graded version newer than the database that queues, or a worker job that is not dequeued as failed.
• A version-behind job in the worker that leaves its email status without a failed entry.
• A job for a workflow missing from both cache and database that the failed-jobs counter misses.
• A burst of sends after one lost reload that makes more than one primary read, or a later send that reads the primary again.
• A 503 body from cdp-api that carries a public code: it stays the internal {error} shape of cdp-api's other 503s.
• Passing before and after, pinning today's behavior: a matching version that reads the primary, a send without the field, and a keyed retry graded at a newer version that doesn't replay.

Proofs at 5d66b71:

• Lane gate: typescript:check, eslint, oxlint and prettier pass. The changed test files pass: 295 parallel and 174 serial tests.
• Shadow diff: the three changed test files at HEAD, run against feat(cdp): key a code send on the sender the caller chose #580's head 64d8940, move nine outcomes, each named by the goal: the version floor in cdp-api and the worker, the refusal when the database is older, the graded version on the queued run, the shared primary reads, and the new failed-jobs count. No test that passes on base fails on HEAD. Evidence is saved with fleet proof.
• Live and cross-version runs: owed at this head. Lane dj3 of map Map: send code-managed email templates with one PostHog call #461 runs the live stack. Not run here.
• Prod shape: a gap until read-only production access exists on 2026-11-02.

Proofs at the pre-rebase heads 2bdc74b and ae46ab4

Live runs saved with fleet proof, curl into maildev. A lost reload is simulated in SQL: the workflow's version and subject change with no reload published.

Case	Both new (2bdc74b, Django #585 4d76e98)	Node at #580, Django #585	Django before #585, Node new	cdp-api new, workers at #580
Send by key and hash, caches warm	202, delivered	202, delivered	202, delivered	202, delivered
Lost reload, then a send by key and hash	202, delivers the new content	202, delivers the cached content, as today	202, delivers the cached content, as today	202, delivers the cached content, as today
Lost reload, then a send by template id	202, delivers the new content	not run	not run	not run
Keyed send before the lost reload, retried after it	202 replayed: true, one email in total	not run	not run	not run
Five sends at once after a lost reload	each 202 and delivers the new content; cdp-api makes one primary read	not run	not run	not run
Rendered-content send	202, delivered	202, delivered	202, delivered	202, delivered
Template by id (control)	202, delivered	202, delivered	202, delivered	202, delivered
Version-behind 503s	none	–	–	–

• The read count comes from cdp-api's Postgres query metric for primary reads of workflows. It rose by 1 across the five sends.
• This head, ae46ab4, differs from 2bdc74b only by the 503 body. No case above reaches that 503.
• Each mixed window delivers the cached copy, which is today's behavior: the older side drops or ignores the version.

CI: the fork has no Depot runners, so those jobs never start, and no secret for the project-board app, so flags-project-board fails; the other GitHub-hosted checks pass. The lane check passes: against master the PR changes only nodejs/ (cross_lane=false).

Release status

• This change is behind a feature flag and is not available to users

Docs update

None.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Agent: Claude Code, Claude Opus 5.5 (claude-opus-5-5)

• Built as lane srv2 of map Map: send code-managed email templates with one PostHog call #461, for decision D3.
• Restacked as lane node3 on 2026-10-31.
• Skills: tdd, writing-tests, stacking-prs, modernize-pr, pr-description, writing-pr-descriptions.

Question	Answer	Who	Where
Does a template send deliver what Django graded?	Yes, through a new Node-chain PR after #580	👤 Human	map #461, decision D3 of 2026-10-25
May the code carry comments?	No; names carry the reasons	👤 Human	map #461, 2026-10-25
What if even the primary is older?	Send nothing and answer the contract's 503 delivery_unavailable with the version lag in detail and why; replaces a new public workflow_version_behind code	👤 Human	approved (conductor, 2026-10-26)
Key the cache on the hash, or on a version floor?	A version floor, because Node has no hash and the worker decides delivery; replaces D3's "key the cache on the hash"	👍 Accepted default	approved (conductor, 2026-10-26)
Bump the workflow version on the asset-URL rewrite in #571?	No; keep it a named known limit, because the reorder makes the dry run and the real run report different totals	👍 Accepted default	approved (conductor, 2026-10-26)
Warm the test caches through the public surface?	Not in this PR; the warm-up reads getHogFlow by name, so a rename fails loudly, and the public warm-up is a follow-up	🤖 Agent	qa-swarm round 2, 2026-11-01
Record a version-behind job's email status?	Yes, as failed, the way the worker records a workflow that is gone	🤖 Agent	map #461 lane node3, review of 2026-10-31

Involvement: one decision from Michael and one from the map's conductor; review round 1 (GPT-6.1 Sol, Opus fresh eyes) fixed, then final reads by GPT-6.1 Sol; after the restack, one GPT-6.1 Sol review (one status finding, fixed in f9abed0) and a final read; a full two-family panel on 2026-11-01 (8 lanes: 1 fixed, 1 rejected, shadow diff owed and run) and a narrow round (fresh eyes: description fixes); pre-rebase live and cross-version proof kept above.

🤖 Generated with Claude Code

[Silthus]

Moved to PostHog#127130.