[Silthus]

Moved from Silthus#584. Part of the code-first rebuild of the email send stack; new in the rebuild, with no closed PR to replace: a template send delivers the content Django graded (decision D3 on map Silthus/posthog#461).

Problem

• A template send on send/email/ right after an upload can deliver the previous content, while Django answered that the caller's template.hash is current.
• Django grades the hash against Postgres. The email goes out from the workflow copy that cdp-api and the hogflow worker cached. A lost or late reload leaves that copy behind for up to two minutes.
• This is the cdp-api and worker half. The Django half, which sends the version, is feat(messaging): send the workflow version a template send by key was checked against #127188.
• Impact: Pays into Grow Workflows through billed workflows_emails actions per paying org: a developer who uploads a template and sends it at once gets the new content, so the first send from code is right. In scope: CDP email sends.

Changes

Warning

🔴 Merge risk: high · two-way door: a revert sends from the cached copy again, as today

• A version check on the CDP email delivery path changes which jobs run, where a wrong refusal drops an email in silence, so the grade is high even behind the flag.
• Shared hot path: the hogflow worker loads flows for every workflow run. The new branch runs only when state.send.hogFlowVersion is set, which only Django from feat(messaging): send the workflow version a template send by key was checked against #127188 sets, behind workflows-email-from-code.
• Every mixed window behaves as today. An older cdp-api strips the unknown field. An older worker ignores it. An older Django doesn't send it. The cross-version runs below show all three.
• Land after feat(cdp): key a code send on the sender the caller chose #127129. No order against feat(messaging): send the workflow version a template send by key was checked against #127188 is needed. The fix works once feat(messaging): send the workflow version a template send by key was checked against #127188 and this PR's workers are both live.

Kind	Files	Lines
Production	5	+101 −12
Tests	3	+214 −0

• A template send uses a workflow version at least as new as the one Django graded. The accepted send can carry hogFlowVersion. When the cached copy is older, cdp-api and the hogflow worker read the workflow from the primary.
• One primary read per workflow serves a burst. Concurrent sends after one upload share one read, and its fresh copy goes into the cache. An older replica load cannot replace it.
• If even the primary is older, nothing is sent. cdp-api queues nothing and answers Django with an internal 503. Django answers the caller with the contract's 503 delivery_unavailable, and puts the version lag in detail and why (feat(messaging): send the workflow version a template send by key was checked against #127188).
• The reason stays internal. workflow_version_behind appears only in the body cdp-api sends to Django, in the metric label and in the log. No public answer carries it.
• A queued job in that state closes as a failed run in the worker. The worker first writes the run's failed lifecycle row durably, with error kind workflow_version_behind, as the hog function worker does for a job it drops. Then the failed result goes through the normal pipeline: the failed metric, the run log, the job failure, and a failed email status. Every row keeps the workflow_version_behind kind, so the run can be rerun by that kind. When the durable write fails, the job stays queued for a retry. Versions only go up, so this state should not happen.
• A job whose workflow is gone counts too. The not-found branch now adds to the same failed-jobs counter with reason: 'workflow_unavailable', so the counter covers every job the worker fails before running it.
• feat(cdp): deliver a code send's content on its own transactional email route #127122's shared read for a missing workflow and this PR's version floor share one read path. A send without a version takes a read it joined as it is; a send with a version reads again only when the joined copy is older.
• The version stays out of the idempotency fingerprint, so a keyed retry that Django grades after an upload still replays.
• Why a version floor and not a hash key: Node has no copy of the content hash. The worker, not cdp-api, decides which content goes out. The template save writes the content and the bumped workflow version in one transaction. So "at least the graded version" means "the graded content or newer", and both sides already have version. An upload that lands between the grade and the send delivers the newer content. It is never older.
• Limit: the floor covers versioned writes. The asset-URL rewrite command, as fix(messaging): sync the transactional workflow on asset url rewrites #118965 reworks it, rewrites a transactional workflow's content before its template sync runs, so the workflow version doesn't bump.
• Running the template pass first bumps the version. But then the real run reports fewer changed rows than the dry run, and its output doesn't say why. So the rewrite stays a known exception, and a send right after one can still deliver the cached copy.
• Rebased onto the reworked feat(cdp): key a code send on the sender the caller chose #127129 and master 43333f8. It now builds on feat(cdp): deliver a code send's content on its own transactional email route #127122 and feat(cdp): key a code send on the sender the caller chose #127129 only. Master's fix(cdp): answer transactional email sends without waiting on kafka #125444 (answer before the Kafka flush) and fix(workflows): drop suppressed cc/bcc instead of skipping the email #118456 (email status in the worker) are under it.

How did you test this code?

Test seams: cdp-api's accepted-send route over HTTP (cdp-api.serial.test.ts), processInvocations on the hogflow worker, processBatch on the email worker with the produced rows, and the hogflow manager service for the read count.

Test rationale, each run red first:

• A send whose graded version is newer than the cached copy that delivers the cached content, in cdp-api and in the worker.
• A graded version newer than the database that cdp-api queues.
• A version-behind job in the worker that leaves the run running, loses its metrics or email status, or gets a later row that replaces workflow_version_behind with hog_error.
• A version-behind job whose failure cannot be recorded, which the worker drops instead of keeping for a retry.
• A job for a workflow missing from both cache and database that the failed-jobs counter misses.
• A burst of sends after one lost reload that makes more than one primary read, or a later send that reads the primary again.
• A 503 body from cdp-api that carries a public code: it stays the internal {error} shape of cdp-api's other 503s.
• Passing before and after, pinning today's behavior: a matching version that reads the primary, a send without the field, and a keyed retry graded at a newer version that doesn't replay.

Proofs at 2b99456, all saved with fleet proof:

• Lane gate: the Node typecheck and the changed test files pass, and the pre-push preflight passes.
• Live run, curl into maildev, Django from feat(messaging): send the workflow version a template send by key was checked against #127188 (which sends the graded version) against this head and against feat(cdp): key a code send on the sender the caller chose #127129's head:
  • A send graded above the stored workflow version: this head answers Django with the internal 503 and sends nothing. feat(cdp): key a code send on the sender the caller chose #127129's head queues it and delivers the older workflow.
  • A template send with a graded version, a code send from the default sender, and a keyed retry after the default sender changes all behave the same on both heads.
• Cross-version run: Django at feat(messaging): send an email with inline html from code #127123's head and at master both send through this head and deliver once. Neither sends the version, so a missing version still means no minimum.
• Shadow diff: the four changed test files at HEAD, run against feat(cdp): key a code send on the sender the caller chose #127129's head 64d8940, move only outcomes named by the goal: the version floor in cdp-api and the worker, the refusal when the database is older, the failed run in the worker, the graded version on the queued run, the shared primary reads, and the new failed-jobs count. No test that passes on base fails on HEAD.
• Prod shape: the read-only check at 5d66b71 still applies. The later commits change only how the worker closes a version-behind job, and read nothing new from storage.
• CI report: the Python and TypeScript duplication warnings name only files outside this diff, which changes only nodejs/. They come from master changes that the stacked base branch does not have yet.

Proofs at the pre-rebase heads 2bdc74b and ae46ab4

Live runs saved with fleet proof, curl into maildev. A lost reload is simulated in SQL: the workflow's version and subject change with no reload published.

Case	Both new (2bdc74b, Django Silthus/posthog#585 (now #127188) 4d76e98)	Node at #127129, Django Silthus/posthog#585 (now #127188)	Django before Silthus/posthog#585 (now #127188), Node new	cdp-api new, workers at #127129
Send by key and hash, caches warm	202, delivered	202, delivered	202, delivered	202, delivered
Lost reload, then a send by key and hash	202, delivers the new content	202, delivers the cached content, as today	202, delivers the cached content, as today	202, delivers the cached content, as today
Lost reload, then a send by template id	202, delivers the new content	not run	not run	not run
Keyed send before the lost reload, retried after it	202 replayed: true, one email in total	not run	not run	not run
Five sends at once after a lost reload	each 202 and delivers the new content; cdp-api makes one primary read	not run	not run	not run
Rendered-content send	202, delivered	202, delivered	202, delivered	202, delivered
Template by id (control)	202, delivered	202, delivered	202, delivered	202, delivered
Version-behind 503s	none	–	–	–

• The read count comes from cdp-api's Postgres query metric for primary reads of workflows. It rose by 1 across the five sends.
• This head, ae46ab4, differs from 2bdc74b only by the 503 body. No case above reaches that 503.
• Each mixed window delivers the cached copy, which is today's behavior: the older side drops or ignores the version.

Release status

• This change is behind a feature flag and is not available to users

Docs update

• None.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Agent: Claude Code, Claude Opus 5.5 (claude-opus-5-5)

• Built as lane srv2 of map Map: send code-managed email templates with one PostHog call Silthus/posthog#461, for decision D3.
• Restacked as lane node3 on 2026-10-31.
• Skills: tdd, writing-tests, stacking-prs, modernize-pr, pr-description, writing-pr-descriptions.

Question	Answer	Who	Where
Does a template send deliver what Django graded?	Yes, through a new Node-chain PR after #127129	👤 Human	map Silthus/posthog#461, decision D3 of 2026-10-25
May the code carry comments?	No; names carry the reasons	👤 Human	map Silthus/posthog#461, 2026-10-25
What if even the primary is older?	Send nothing and answer the contract's 503 delivery_unavailable with the version lag in detail and why; replaces a new public workflow_version_behind code	👤 Human	approved (conductor, 2026-10-26)
Key the cache on the hash, or on a version floor?	A version floor, because Node has no hash and the worker decides delivery; replaces D3's "key the cache on the hash"	👍 Accepted default	approved (conductor, 2026-10-26)
Bump the workflow version on the asset-URL rewrite in #118965?	No; keep it a named known limit, because the reorder makes the dry run and the real run report different totals	👍 Accepted default	approved (conductor, 2026-10-26)
Warm the test caches through the public surface?	Not in this PR; the warm-up reads getHogFlow by name, so a rename fails loudly, and the public warm-up is a follow-up	🤖 Agent	qa-swarm round 2, 2026-11-01
How does the worker close a version-behind job?	As a failed run: a durable workflow_version_behind lifecycle row first, then the failed result through the normal pipeline; replaces recording only the email status	🤖 Agent	qa-swarm rounds 1 and 5 and final reads, 2026-11-02

Involvement: one decision from Michael and one from the map's conductor; review round 1 (GPT-6.1 Sol, Opus fresh eyes) fixed, then final reads by GPT-6.1 Sol; after the restack, one GPT-6.1 Sol review (one status finding, fixed in f9abed0) and a final read; a full two-family panel on 2026-11-01 (8 lanes: 1 fixed, 1 rejected, shadow diff owed and run) and a narrow round (fresh eyes: description fixes); after the move, a GPT-6.1 Sol correctness read of the whole PR at 5d66b71, clean; a further review round on the worker's lifecycle row, fixed, and GPT-6.1 Sol final reads until one was clean at 2b99456; live, cross-version and shadow-diff proof at 2b99456; pre-rebase live and cross-version proof kept above.

🤖 Generated with Claude Code

[github-actions]

🤖 CI report

✅ Trunk lane — non-backend lane

This PR is assigned to the non-backend lane. It does not run backend Python tests and may merge in parallel with PRs in other lanes.

✅ Duplication (Python) — clean

New Python code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying.

✅ Duplication (TypeScript) — clean

New TypeScript code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying.

❌ Hogbox preview — build failed

The preview didn't come up for commit f9b0d6f. See the build log for the failing step. It'll retry on the next push.

_{Previews are optional and never block merging. A failure here is often a hogland or tailnet hiccup rather than anything in your PR, so the check stays green and this section is the status.}

[posthog]

PostHog Review · Standard · reviewed 5d66b71: 1 Should fix.

This is the only automatic review while this pull request is a draft. Comment @posthog review to review new changes sooner. Otherwise, a review of the new changes runs when you mark it ready for review.

See this review and its findings in PostHog Review. Comment @posthog review --deep for a Deep review. A Deep review takes longer and looks deeper than a Standard review.

[posthog]

P2 · Record a terminal invocation failure before dequeuing

When the primary workflow is behind send.hogFlowVersion, this branch removes the job without producing a terminal invocation lifecycle row. queueUnavailableRun updates only email status, leaving the API's existing running lifecycle row unchanged and the run permanently unavailable for rerun. Durably record the terminal failure before dequeuing, as loadHogFunctions does.

[trunk-io]

Merging to master in this repository is managed by Trunk.

• To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here