Repository navigation
A job's owner carries the lease generation it was written under - #3249
Merged
FlexOr2 merged 22 commits intoNov 17, 2026
Merged
Conversation
Lapse cleanup and the reconcile pass failed a job when its owner's lease read GONE. A pass that read GONE, followed by a renewal and new jobs of the same owner, failed those new jobs: jobs carried only owner_id, so the duty of tandem's README "fence effects by generation" was not met for durable job ownership (split off #3209). jobs.owner_generation (expand-only migration, nullable) is written together with owner_id by one function, db.queries.jobs.owner_columns, from the process's JobOwner (its id and the Hold of its lease), for every writer: the chat turn, the cover claim, and the generation, LoRA, scoring and model-load jobs. The pass reads the (owner, generation) stamps of the active jobs before it reads the leases; a stamp whose generation is not the live lease's reads GONE, and a job written after the read names a stamp the pass does not know. Rows without a generation keep being judged by their owner alone. Rejected: failing by Lapse.generation alone in the lapse handler, which would leave the reconcile pass and the rate-limit recovery as a second decider. Work-Item: #3227 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ze ceilings hold The new migration's tests were a near copy of the owner_id migration's, so both now run in one parametrized module. The scoring cases of test_jobs repeat one call, now one helper, and a few split statements in the files the lane touched are joined, so no file grows past its ceiling. Work-Item: #3227 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…en its read and its update A test at the lifecycle layer drives fail_jobs_of_dead_owners through a Leases port whose state() answers GONE, then ends and renews the owner and commits a job of the new generation before the pass updates: the old generation's job fails and the new one keeps running. The lapse-loop test seeds its job with the lease's real generation and keeps one case for a row that names none. Work-Item: #3227 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…neration Every production writer stamps a generation, so the crash, clean-stop and music-worker cases now drive the stamped path. A previous release's jobs and the restart-empty case keep rows that name no generation, which are judged by their owner alone. Work-Item: #3227 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…itted before cleanup reads jobs The test held B's reply to its read of A's lapsed lease, locked the jobs table so B's pass waits at its first job read, let A start its lease again and commit a job of the new generation, then released the lock: only the old generation's job fails. The proxy holds one reply after the server ran the command; each process of a case may reach Redis through a line of its own. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ike the rate-limit recovery does Both paths read active_job_owner_stamps and owner_lease_liveness; the reaper no longer drops its own owner id, so a job of a generation this process's lease lost fails (tandem's host duty) and one of its current generation keeps running. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… until the new job is committed Both of a process's cleanup loops read the lapsed owner's lease, and the proxy held one reply, so the other pass could end the old job early and the test read the new job before a pass that read the old lease had judged it. The proxy now holds every reply to the old key; the test starts the new generation and commits its job while both are held, so only held passes can fail the old job, each in one transaction with its verdict on the new one. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…; the legacy case says it is about jobs without a generation Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… jobs and what an empty restart does to a live owner's stamped jobs Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ils when a held read expired Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…e timeout The proxy learned that a client gave up only from a flag another thread set after the client's close arrived, so a client that had timed out could still see its reply counted as delivered. A reply held for as long as the client waits is now expired whatever the line did, so the proof of the late job cannot be satisfied by a retried read or a later pass. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ker tests The test called _worker_on with a monkeypatch argument that the helper no longer takes after the merge of main, so shard 2 failed and no coverage artifact reached sonar. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… that fails jobs The proxy-timed two-process proof raced the client's socket timeout, retries and later passes, so no counter of the proxy could show which pass ended the old job. The race is now driven where it is deterministic: the lease read answers GONE, the owner renews and commits a new-generation job, and only then the pass updates, once through the cleanup call, once through the lapse loop and once through the reaper loop. The proxy's hold machinery and the helpers only it used are removed; it stays recoverable from git. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…e legacy row The test polled PostgreSQL on a 10 ms loop interval and discarded the pass it cancelled. It now runs each pass once, the loops stopped at the pause after it, and fails when the pass fails: the cleanup call raises, the loops show it in their health. The scenario also writes a job that names no generation, which fails with the lapsed generation while the new generation's job keeps running. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…erati Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
FlexOr2
deleted the
claude/issue-3227-a-job-s-owner-carries-its-lease-generati
branch
November 17, 2026 22:54
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Work-Item: #3227
Closes #3227
Routing: build sonnet/high; fix sonnet/high; review opus/medium; test sonnet/high; gate gpt-6-astra/high; land sonnet/medium
A job's owner now carries the lease generation it was written under, so lapse cleanup and the reconcile pass fail only the jobs of the lapsed generation (tandem's host duty "fence effects by generation", the part #3209's slice 1 deferred).
What changes
e5a2c8f1d703(expand-only): nullablejobs.owner_generation.worker_liveness.JobOwner(owner id and the leaseHold, whose generation may change) is whatkeep_owner_leasenow yields; web keeps it inAppContext.owner, workers inWorkerBase.owner.db.queries.jobs.owner_columnsstampsowner_idandowner_generationtogether forcreate_job(chat turn),claim_next_cover_job, andupdate_job_status(generation, LoRA, scoring, model load and download viaJobStatusUpdate.owner).(owner, generation)stamps of active jobs first (active_job_owner_stamps), thenLeases.state;owner_lease_livenessreads a stamp whose generation is not the live lease's as GONE. A job written after the stamps were read names a stamp the pass does not know and is left alone. The rate-limit recovery increate_job_with_rate_limituses the same stamps, this process's own included: the decision "own jobs are judged like any other's" lives inactive_job_owner_stamps, which both paths call without filtering (tandem's host duty: the jobs of a lapsed generation fail, the own process included; no stated reason to exempt it). A job of a generation this process's lease lost fails although the process lives on; one of its current generation keeps running (test_a_job_of_a_generation_this_process_lost_fails_and_its_current_ones_keep_running).docs/architecture.mdowner liveness updated.Evidence
test_a_job_the_owner_wrote_after_its_lease_was_read_gone_keeps_running_while_the_old_one_failsruns each pass that fails jobs (the cleanup call, the lapse loop and the reaper loop) through a Leases port whose firststate()reads GONE and then renews the owner and commits a new-generation job before the pass updates. The old job fails and the new one keeps running in all three cases; the cases fail (3 of 3) when the verdict judges by owner alone. A proxy-timed two-process proof was dropped: its held replies raced the client's socket timeout, retries and later passes, so it could not show which pass ended the old job.test_a_redis_that_restarts_empty_ends_the_jobs_that_name_the_generation_it_lost).test_only_the_jobs_of_a_lapsed_generation_fail_and_a_later_start_keeps_its_jobs(a job written after the stamps were read survives), stamp reads intest_worker_liveness_owner_lease.py, the worker'sownergeneration intest_worker_base.py, the migration cases parametrized over both owner migrations.scripts/test_postgresql.sh: 84, 249+, 892+, 623 and 733 passed across the lifecycle, jobs, db, api, worker, cover runner and owner-recovery files;bash scripts/static_checks.shexit 0.Remaining risk
A Redis that restarts empty gives a live owner's lease a new generation on its next renewal, so the running jobs that name the old one are then failed by the next pass (rows without a generation are not). A
JobOwnerread in the microseconds between a lease renewal and the stamp can write the previous generation; the live process's job is then failed by the next pass. A job running in a process whose lease lapsed fails (the duty), though the process lives on.🤖 Generated with Claude Code