Skip to content

fix(jobqueue): skip payloads completed during send - #252

Open
bajtos wants to merge 2 commits into
mainfrom
claude/heuristic-cohen-77c5eb
Open

bajtos wants to merge 2 commits into
mainfrom
claude/heuristic-cohen-77c5eb

Conversation

@bajtos

@bajtos bajtos commented Nov 20, 2026

Copy link
Copy Markdown
Member

Written by Claude.

Fix a race in the dedup job queue that let Postgres run a completed job a second time. The race also made TestJobQueue_DedupPreventsDuplicates/dedup/postgres flaky (for example this run).

Root cause

sendAndGetIDTx checks job_done, then inserts into jobs with ON CONFLICT DO NOTHING. These are two statements, and under Postgres READ COMMITTED each takes its own snapshot. The worker's Delete removes the live job and writes job_done in one transaction. When that transaction commits between the two statements, the check sees no job_done row and the insert finds no live job, so the payload is queued and handled again. If the insert starts while the delete is still open, it waits for the delete to commit and then inserts anyway.

The test hits this window because the worker increments the handler counter before it calls Delete. require.Eventually(processed == 1) returns while the delete is still pending, and the third Enqueue races with it. SQLite serialises writers, so the SQLite variant never sees the interleaving.

The JSON serializer cannot be used with dedupe queue log line is unrelated. TestMessage is a struct, encoding/json marshals struct fields in a fixed order, and the SQLite variant uses the same serializer and passes.

Fix

After a successful insert, check job_done again in the same transaction and delete the new row if the payload completed in the meantime. The insert can only succeed once the live job's delete has committed, and that same commit wrote job_done, so the second check sees it. The new row is still uncommitted, so no worker can have received it. The fix relies on READ COMMITTED: a caller of SendTx running REPEATABLE READ or SERIALIZABLE keeps its earlier snapshot, and the code comment says so.

TestQueue_DedupSkipsPayloadCompletedDuringSend reproduces the interleaving on Postgres. It holds a delete transaction open, starts Send for the same payload, commits the delete, and asserts that nothing is left to receive.

Testing

go test -count=20 ./lib/jobqueue/... passes locally. Docker was unavailable there, so the Postgres variants skipped and CI covers them.

On Postgres, the dedup queue could run a completed job a second time.
sendAndGetIDTx checks job_done and then inserts into jobs in two
statements, and under READ COMMITTED each statement takes its own
snapshot. When the worker's Delete of the same payload commits between
them, the check sees no job_done row and the insert finds no live job
(or waits for the delete and then proceeds), so the payload is queued
and handled again. SQLite serialises writers, so it never hit this.

This made TestJobQueue_DedupPreventsDuplicates/dedup/postgres flaky:
the handler counter reaches 1 before the worker calls Delete, so the
test's third Enqueue races with that Delete.

Check job_done again after a successful insert and remove the new row
when the payload completed in the meantime. The insert can only succeed
once the live job's delete has committed, and that transaction also
wrote job_done, so the second check sees it.

Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>
Assisted-by: Claude:claude-opus-5-5
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-20T16:25:20.048979Z ab324c5 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant