Skip to content

A job's owner carries the lease generation it was written under - #3249

Merged
FlexOr2 merged 22 commits into
mainfrom
claude/issue-3227-a-job-s-owner-carries-its-lease-generati
Nov 17, 2026
Merged

FlexOr2 merged 22 commits into
mainfrom
claude/issue-3227-a-job-s-owner-carries-its-lease-generati

Conversation

@FlexOr2

@FlexOr2 FlexOr2 commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Work-Item: #3227
Closes #3227

Routing: build sonnet/high; fix sonnet/high; review opus/medium; test sonnet/high; gate gpt-6-astra/high; land sonnet/medium

A job's owner now carries the lease generation it was written under, so lapse cleanup and the reconcile pass fail only the jobs of the lapsed generation (tandem's host duty "fence effects by generation", the part #3209's slice 1 deferred).

What changes

  • Migration e5a2c8f1d703 (expand-only): nullable jobs.owner_generation.
  • worker_liveness.JobOwner (owner id and the lease Hold, whose generation may change) is what keep_owner_lease now yields; web keeps it in AppContext.owner, workers in WorkerBase.owner.
  • One writer: db.queries.jobs.owner_columns stamps owner_id and owner_generation together for create_job (chat turn), claim_next_cover_job, and update_job_status (generation, LoRA, scoring, model load and download via JobStatusUpdate.owner).
  • One reader: the pass reads the (owner, generation) stamps of active jobs first (active_job_owner_stamps), then Leases.state; owner_lease_liveness reads a stamp whose generation is not the live lease's as GONE. A job written after the stamps were read names a stamp the pass does not know and is left alone. The rate-limit recovery in create_job_with_rate_limit uses the same stamps, this process's own included: the decision "own jobs are judged like any other's" lives in active_job_owner_stamps, which both paths call without filtering (tandem's host duty: the jobs of a lapsed generation fail, the own process included; no stated reason to exempt it). A job of a generation this process's lease lost fails although the process lives on; one of its current generation keeps running (test_a_job_of_a_generation_this_process_lost_fails_and_its_current_ones_keep_running).
  • Rows with NULL generation (written before the migration) and old-key owners keep today's behaviour (judged by owner alone).
  • docs/architecture.md owner liveness updated.

Evidence

  • The ruled race (a GONE read, then a renewal and a new job, then the update) is pinned at the lifecycle layer with a real database and no timing: test_a_job_the_owner_wrote_after_its_lease_was_read_gone_keeps_running_while_the_old_one_fails runs each pass that fails jobs (the cleanup call, the lapse loop and the reaper loop) through a Leases port whose first state() reads GONE and then renews the owner and commits a new-generation job before the pass updates. The old job fails and the new one keeps running in all three cases; the cases fail (3 of 3) when the verdict judges by owner alone. A proxy-timed two-process proof was dropped: its held replies raced the client's socket timeout, retries and later passes, so it could not show which pass ended the old job.
  • The two-process crash, clean-stop and music-worker cases and the lapse-loop case seed jobs under the owner's real lease generation; rows without a generation keep a lapse-loop case and the previous-release case; a Redis that restarts empty has two cases: jobs without a generation follow their owner (only the ghost's fail), and jobs that name the lost generation fail once the lease starts anew (test_a_redis_that_restarts_empty_ends_the_jobs_that_name_the_generation_it_lost).
  • test_only_the_jobs_of_a_lapsed_generation_fail_and_a_later_start_keeps_its_jobs (a job written after the stamps were read survives), stamp reads in test_worker_liveness_owner_lease.py, the worker's owner generation in test_worker_base.py, the migration cases parametrized over both owner migrations.
  • Targeted runs via scripts/test_postgresql.sh: 84, 249+, 892+, 623 and 733 passed across the lifecycle, jobs, db, api, worker, cover runner and owner-recovery files; bash scripts/static_checks.sh exit 0.

Remaining risk

A Redis that restarts empty gives a live owner's lease a new generation on its next renewal, so the running jobs that name the old one are then failed by the next pass (rows without a generation are not). A JobOwner read in the microseconds between a lease renewal and the stamp can write the previous generation; the live process's job is then failed by the next pass. A job running in a process whose lease lapsed fails (the duty), though the process lives on.

🤖 Generated with Claude Code

FlexOr2 and others added 22 commits November 16, 2026 18:41
Lapse cleanup and the reconcile pass failed a job when its owner's lease
read GONE. A pass that read GONE, followed by a renewal and new jobs of the
same owner, failed those new jobs: jobs carried only owner_id, so the duty of
tandem's README "fence effects by generation" was not met for durable job
ownership (split off #3209).

jobs.owner_generation (expand-only migration, nullable) is written together
with owner_id by one function, db.queries.jobs.owner_columns, from the
process's JobOwner (its id and the Hold of its lease), for every writer: the
chat turn, the cover claim, and the generation, LoRA, scoring and model-load
jobs. The pass reads the (owner, generation) stamps of the active jobs before
it reads the leases; a stamp whose generation is not the live lease's reads
GONE, and a job written after the read names a stamp the pass does not know.
Rows without a generation keep being judged by their owner alone.

Rejected: failing by Lapse.generation alone in the lapse handler, which would
leave the reconcile pass and the rate-limit recovery as a second decider.

Work-Item: #3227

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ze ceilings hold

The new migration's tests were a near copy of the owner_id migration's, so
both now run in one parametrized module. The scoring cases of test_jobs
repeat one call, now one helper, and a few split statements in the files the
lane touched are joined, so no file grows past its ceiling.

Work-Item: #3227

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…en its read and its update

A test at the lifecycle layer drives fail_jobs_of_dead_owners through a Leases
port whose state() answers GONE, then ends and renews the owner and commits a job of
the new generation before the pass updates: the old generation's job fails and the new
one keeps running. The lapse-loop test seeds its job with the lease's real generation
and keeps one case for a row that names none.

Work-Item: #3227

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…neration

Every production writer stamps a generation, so the crash, clean-stop and music-worker
cases now drive the stamped path. A previous release's jobs and the restart-empty case
keep rows that name no generation, which are judged by their owner alone.

Work-Item: #3227

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…itted before cleanup reads jobs

The test held B's reply to its read of A's lapsed lease, locked the jobs table so B's pass
waits at its first job read, let A start its lease again and commit a job of the new
generation, then released the lock: only the old generation's job fails. The proxy holds
one reply after the server ran the command; each process of a case may reach Redis through
a line of its own.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ike the rate-limit recovery does

Both paths read active_job_owner_stamps and owner_lease_liveness; the reaper
no longer drops its own owner id, so a job of a generation this process's
lease lost fails (tandem's host duty) and one of its current generation keeps
running.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… until the new job is committed

Both of a process's cleanup loops read the lapsed owner's lease, and the proxy held
one reply, so the other pass could end the old job early and the test read the new
job before a pass that read the old lease had judged it. The proxy now holds every
reply to the old key; the test starts the new generation and commits its job while
both are held, so only held passes can fail the old job, each in one transaction
with its verdict on the new one.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…; the legacy case says it is about jobs without a generation

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… jobs and what an empty restart does to a live owner's stamped jobs

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ils when a held read expired

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…e timeout

The proxy learned that a client gave up only from a flag another thread set
after the client's close arrived, so a client that had timed out could still
see its reply counted as delivered. A reply held for as long as the client
waits is now expired whatever the line did, so the proof of the late job
cannot be satisfied by a retried read or a later pass.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ker tests

The test called _worker_on with a monkeypatch argument that the helper no longer takes
after the merge of main, so shard 2 failed and no coverage artifact reached sonar.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… that fails jobs

The proxy-timed two-process proof raced the client's socket timeout, retries and later
passes, so no counter of the proxy could show which pass ended the old job. The race is now
driven where it is deterministic: the lease read answers GONE, the owner renews and commits
a new-generation job, and only then the pass updates, once through the cleanup call, once
through the lapse loop and once through the reaper loop. The proxy's hold machinery and the
helpers only it used are removed; it stays recoverable from git.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…e legacy row

The test polled PostgreSQL on a 10 ms loop interval and discarded the pass
it cancelled. It now runs each pass once, the loops stopped at the pause
after it, and fails when the pass fails: the cleanup call raises, the loops
show it in their health. The scenario also writes a job that names no
generation, which fails with the lapsed generation while the new
generation's job keeps running.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…erati

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

@FlexOr2
FlexOr2 merged commit 9dc4e6e into main Nov 17, 2026
33 checks passed
@FlexOr2
FlexOr2 deleted the claude/issue-3227-a-job-s-owner-carries-its-lease-generati branch November 17, 2026 22:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A job's owner carries its lease generation, so lapse cleanup fails only the lapsed hold's jobs

1 participant