Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
0920559
A job's owner carries the lease generation it was written under
FlexOr2 Nov 16, 2026
402dc1e
Job-owner tests share one scoring helper and one migration module; si…
FlexOr2 Nov 16, 2026
9502549
Merge origin/main into the lane
FlexOr2 Nov 16, 2026
d3f6363
The lifecycle pass is proven against an owner that starts again betwe…
FlexOr2 Nov 16, 2026
e760fc0
The two-process owner tests seed jobs under the owner's real lease ge…
FlexOr2 Nov 16, 2026
f165ef0
Merge origin/main into the lane
FlexOr2 Nov 16, 2026
d88176d
Merge origin/main into the lane
FlexOr2 Nov 16, 2026
6f50259
The lease-renewal race is proven with two processes, the new job comm…
FlexOr2 Nov 16, 2026
0f2178e
A job owner of a live lease always names the generation the test reads
FlexOr2 Nov 16, 2026
7b41a56
The owner reaper judges this process's own jobs by their generation l…
FlexOr2 Nov 16, 2026
aa25d93
The lease-renewal race holds every pass that reads the old generation…
FlexOr2 Nov 16, 2026
fecb0d7
An empty Redis restart ends the jobs that name the generation it lost…
FlexOr2 Nov 16, 2026
c292049
Merge origin/main into the lane
FlexOr2 Nov 16, 2026
9e0a9b7
The architecture document says the job pass judges this process's own…
FlexOr2 Nov 16, 2026
10f08a5
The held-read proof counts the replies that reached their pass and fa…
FlexOr2 Nov 16, 2026
39011d1
A held reply counts as expired when it was held for the client's whol…
FlexOr2 Nov 16, 2026
5ead7d4
The worker lease-generation test starts its worker like the other wor…
FlexOr2 Nov 17, 2026
6d45ae3
The cleanup race is proven at the PostgreSQL level through every pass…
FlexOr2 Nov 17, 2026
ce31b49
The stale-GONE race test drives one pass per owner pass and covers th…
FlexOr2 Nov 17, 2026
292714a
The owner loop test proves its single pass recorded no failure
FlexOr2 Nov 17, 2026
bbbb64e
The owner loop test's return annotation fits the line length
FlexOr2 Nov 17, 2026
ff00639
Merge main into claude/issue-3227-a-job-s-owner-carries-its-lease-gen…
FlexOr2 Nov 17, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 15 additions & 9 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -1619,17 +1619,21 @@ apply the same queued-job rule.
(`WorkerBase.owner_id`). A process writes its id on the jobs it runs: a chat job when
the turn starts, a cover job when it claims it, a worker job (generation, LoRA
training, scoring, model load and download) when it first marks it running
(`jobs.owner_id`, nullable; a job that waits in a queue and older rows have none
and get no owner verdict). Its lease is a holder of tandem's `Leases` in the family `owner`,
(`jobs.owner_id` and `jobs.owner_generation`, nullable, written together by
`db.queries.jobs.owner_columns` from the process's `JobOwner`: its id and the generation of its
lease; a job that waits in a queue has none and gets no owner verdict, and a row written before
generations were stored has an owner without a generation and is judged by its owner alone). Its lease is a holder of tandem's `Leases` in the family `owner`,
built by `worker_liveness.owner_leases` for every process from the same spans: a lease lasts
`OWNER_LEASE_TTL_SECONDS` (15 s) and is renewed every `OWNER_LEASE_REFRESH_SECONDS` (5 s) by
`worker_liveness.keep_owner_lease` (`Leases.kept`), in a web process from the end of start-up to
the end of `_lifespan`, in a worker from after its database has opened to its `on_shutdown`. A
web process holds an asyncio client for the leases (`AppContext.async_redis`); the sync client
serves everything else. The `owner_gone_reaper` loop of every web process runs
`lifecycle.fail_jobs_of_dead_owners` every `OWNER_GONE_CHECK_INTERVAL_SECONDS`
(5 s) and once at start: it reads the owners of active jobs through `Leases.state` (`worker_liveness.owner_lease_liveness`)
(`alive`, `gone` or `unknown`) and fails the active jobs of every `gone` owner, whether a web
(5 s) and once at start: it reads the owner stamps of active jobs (`active_job_owner_stamps`: owner and generation) first,
then their leases through `Leases.state` (`worker_liveness.owner_lease_liveness`)
(`alive`, `gone` or `unknown`); a stamp whose generation is not the live lease's reads `gone`,
for that lease lapsed. It fails the active jobs of every `gone` stamp, whether of a web
process or a worker, with the one
restart verdict `JOB_ERROR_SERVER_RESTARTED` (`error_type` `server_restart`), then
removes the suggestion and staging directory of the covers it ended. The `owner_lapses` loop
Expand All @@ -1651,17 +1655,19 @@ panel shows "Try again" at once; after a crash within 20 s.
`owner_gone_reaper` loop health shows in `/health/details`, and fails nothing. A
Redis that lost its data reads an absent lease `unknown` until the lease span has passed
since the first command after the loss (tandem's epoch), by when the pods have renewed
their leases; an owner id the reading lacks is `unknown`, and the job pass never asks about
its own process. A renewal that fails is retried every interval and logged by tandem; the
their leases; an owner id the reading lacks is `unknown`. The job pass judges this process's own jobs
like any other's, so a job of a generation its lease lost fails although the process lives on; after an
empty restart, a live owner's stamped jobs of the lost generation fail on the first pass after its
renewal, while rows without a generation follow their owner. A renewal that fails is retried every interval and logged by tandem; the
`owner_gone_reaper` tick reads this process's own lease, not its old key, and the `owner_lease`
loop health shows in `/health/details` when the roster of live leases lacks it (`owner_lapses` shows the lapse
pass apart). A pod that lost its lease while alive (a Redis outage longer than the
lease span) may see a sibling fail its jobs; every later write of that job then refuses, as
for any terminal job. When its lease starts anew, `keep_owner_lease` sees the changed
generation of tandem's `Hold`, logs it as a start and, in a web process, removes what every
cover whose stored status has ended left behind (`lifecycle.reconcile_ended_covers`). The stored job status is the only fence: a job
carries no lease generation, so a lapse cleanup that read the old lease may still fail the
jobs of the new one. For one release every process also writes `<prefix>:owner:<id>` for the
cover whose stored status has ended left behind (`lifecycle.reconcile_ended_covers`). A job a process wrote under the new
generation names a stamp the pass that read the old one does not know, so it keeps running, and
the stored job status still fences every write to a job that was failed. For one release every process also writes `<prefix>:owner:<id>` for the
previous release's readers and reads an owner whose old key is live as `alive`
(`owner_lease_old_key`). A cover's staging directory is named by its job id alone, so it
is removed also when the album was deleted. A crashed LoRA training leaves its voice in
Expand Down
12 changes: 6 additions & 6 deletions quality/baseline.toml
Original file line number Diff line number Diff line change
Expand Up @@ -44,18 +44,18 @@
"frontend/src/routes/layout.test.ts" = 1395
"frontend/src/routes/settings/users/+page.svelte" = 919
"src/acestep_worker/wrapper.py" = 1057
"src/songmaker_cli/jobs/generation.py" = 1124
"src/songmaker_cli/jobs/lora_training.py" = 1508
"src/songmaker_cli/jobs/generation.py" = 1121
"src/songmaker_cli/jobs/lora_training.py" = 1507
"src/songmaker_cli/queue_streams.py" = 893
"tests/acestep_engine/test_client.py" = 818
"tests/acestep_worker/test_wrapper.py" = 1501
"tests/acestep_worker/test_wrapper_train_lora.py" = 1180
"tests/songmaker_cli/jobs/test_lora_training.py" = 1841
"tests/songmaker_cli/jobs/test_lora_training.py" = 1839
"tests/songmaker_cli/test_admin_api.py" = 999
"tests/songmaker_cli/test_api.py" = 4554
"tests/songmaker_cli/test_db.py" = 2430
"tests/songmaker_cli/test_api.py" = 4553
"tests/songmaker_cli/test_db.py" = 2429
"tests/songmaker_cli/test_db_postgresql.py" = 831
"tests/songmaker_cli/test_jobs.py" = 3071
"tests/songmaker_cli/test_jobs.py" = 3066
"tests/songmaker_cli/test_library_api.py" = 1621
"tests/songmaker_cli/test_lora_api.py" = 1170
"tests/songmaker_cli/test_playlist_api.py" = 1100
Expand Down
18 changes: 10 additions & 8 deletions src/songmaker_cli/api_helpers.py
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@
from songmaker_cli.db.queries.albums import get_album
from songmaker_cli.db.queries.generations import get_generation, get_generations
from songmaker_cli.db.queries.jobs import (
active_job_owner_ids,
active_job_owner_stamps,
count_total_queued_jobs,
count_user_active_jobs,
count_user_jobs_in_window,
Expand All @@ -75,6 +75,8 @@
from songmaker_cli.refusals import Refusal
from songmaker_cli.settings import get_settings
from songmaker_cli.worker_liveness import (
JobOwner,
JobOwnerStamp,
WorkerLiveness,
owner_lease_liveness,
read_worker_liveness,
Expand Down Expand Up @@ -233,7 +235,7 @@ def _enforce_job_limits(
user: AuthenticatedUser,
job_type: JobType,
worker_liveness: WorkerLiveness,
owner_liveness: Mapping[str, Liveness],
owner_liveness: Mapping[JobOwnerStamp, Liveness],
) -> None:
"""Recover stale jobs, then roll back with 429 when a queue or rate limit is reached."""
settings = get_settings()
Expand Down Expand Up @@ -270,11 +272,11 @@ def _enforce_job_limits(


async def _read_owner_liveness_or_unknown(
owner_leases: Leases, owner_ids: set[str],
) -> Mapping[str, Liveness]:
owner_leases: Leases, stamps: set[JobOwnerStamp],
) -> Mapping[JobOwnerStamp, Liveness]:
"""A Redis that cannot be read leaves every owner UNKNOWN, which fails no job."""
try:
return await owner_lease_liveness(owner_leases, owner_ids)
return await owner_lease_liveness(owner_leases, stamps)
except StoreError:
_log.warning("Could not read owner liveness", exc_info=True)
return {}
Expand All @@ -287,7 +289,7 @@ async def create_job_with_rate_limit(
song_id: str | None = None,
*,
version_id: str | None = None,
owner_id: str | None = None,
owner: JobOwner | None = None,
redis: Redis,
owner_leases: Leases,
) -> Job:
Expand All @@ -309,7 +311,7 @@ async def create_job_with_rate_limit(
worker_ids = [worker.id for worker in list_worker_identities(session)]
worker_liveness = read_worker_liveness(redis, worker_ids)
owner_liveness = await _read_owner_liveness_or_unknown(
owner_leases, active_job_owner_ids(session),
owner_leases, active_job_owner_stamps(session),
)
session.commit()
lock_until_transaction_ends(session, AdvisoryLock.RATE_LIMIT)
Expand All @@ -326,7 +328,7 @@ async def create_job_with_rate_limit(

return create_job(
session, job_type, user_id=user.id, song_id=song_id, version_id=version_id,
owner_id=owner_id,
owner=owner,
)


Expand Down
4 changes: 3 additions & 1 deletion src/songmaker_cli/app_context.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
from webauth.rate_limit import RedisRateLimitBackend

from songmaker_cli.constants import REDIS_RL_IP_PREFIX
from songmaker_cli.worker_liveness import owner_leases
from songmaker_cli.worker_liveness import JobOwner, owner_leases

if TYPE_CHECKING:
from collections.abc import Iterator
Expand All @@ -40,6 +40,8 @@ class AppContext:
owner_id: str = field(default_factory=lambda: str(uuid.uuid4()))
"""Names this process on the jobs it runs; its lease in Redis says whether it lives."""
owner_leases: Leases = field(init=False)
owner: JobOwner = field(init=False)
"""This process as the owner of its jobs, set once its lease is held."""

def __post_init__(self) -> None:
self.owner_leases = owner_leases(self.async_redis)
Expand Down
2 changes: 1 addition & 1 deletion src/songmaker_cli/conversation_api.py
Original file line number Diff line number Diff line change
Expand Up @@ -256,7 +256,7 @@ async def _prepare_chat_turn(
user,
JobType.CHAT,
redis=ctx.redis, owner_leases=ctx.owner_leases,
owner_id=ctx.owner_id,
owner=ctx.owner,
)
_refuse_while_another_turn_runs(session, user)
update_job_status(session, job.id, JobStatusUpdate(JobStatus.RUNNING))
Expand Down
7 changes: 4 additions & 3 deletions src/songmaker_cli/cover_runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@
from songmaker_cli.job_types import JobStatus, JobType
from songmaker_cli.jobs._runtime import _touch_heartbeat, _update_job
from songmaker_cli.messages import Message, message
from songmaker_cli.worker_liveness import JobOwner

log = logging.getLogger(__name__)

Expand Down Expand Up @@ -171,7 +172,7 @@ def reconcile_reaped_cover_jobs(db_factory, audio_dir: Path) -> int:
async def run_next_cover_job(app) -> bool:
"""Claim and run one queued cover job."""
ctx = app.state.ctx
job_id = await asyncio.to_thread(_claim_next_cover_job, ctx.db, ctx.owner_id)
job_id = await asyncio.to_thread(_claim_next_cover_job, ctx.db, ctx.owner)
if job_id is None:
return False
await run_claimed_cover_suggestion_job(app, job_id)
Expand Down Expand Up @@ -365,9 +366,9 @@ def build_cover_prompt(album, songs) -> str:
return header + json.dumps(album_data, ensure_ascii=False, separators=(",", ":"))


def _claim_next_cover_job(db_factory, owner_id: str) -> str | None:
def _claim_next_cover_job(db_factory, owner: JobOwner) -> str | None:
with db_factory() as session:
job = claim_next_cover_job(session, owner_id)
job = claim_next_cover_job(session, owner)
session.commit()
return None if job is None else job.id

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
"""jobs name the lease generation of the process that runs them

Revision ID: e5a2c8f1d703
Revises: c7e1a4d9b2f8
Create Date: 2026-11-16 18:00:00.000000

The generation is the value tandem's lease holds for one start of a process: a lease that lapsed
and started anew has another. A row written before this column keeps NULL and is judged by its
owner alone, as before.
"""
from typing import Sequence

import sqlalchemy as sa
from alembic import op

revision: str = 'e5a2c8f1d703'
down_revision: str | Sequence[str] | None = 'c7e1a4d9b2f8'
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None


def upgrade() -> None:
op.add_column('jobs', sa.Column('owner_generation', sa.Text(), nullable=True))


def downgrade() -> None:
op.drop_column('jobs', 'owner_generation')
5 changes: 2 additions & 3 deletions src/songmaker_cli/db/models.py
Original file line number Diff line number Diff line change
Expand Up @@ -106,9 +106,7 @@ class Album(ShareMixin, Base):

deleted_at: Mapped[datetime | None] = mapped_column(TZDateTime, nullable=True)
is_archived: Mapped[bool] = mapped_column(Boolean, default=False)
archived_at: Mapped[datetime | None] = mapped_column(
TZDateTime, nullable=True, index=True,
)
archived_at: Mapped[datetime | None] = mapped_column(TZDateTime, nullable=True, index=True)

songs: Mapped[list[Song]] = relationship(back_populates="album", cascade=DELETE_ORPHANS)
cover_suggestions: Mapped[list[AlbumCoverSuggestion]] = relationship(
Expand Down Expand Up @@ -407,6 +405,7 @@ class Job(Base):
)
worker_pid: Mapped[int | None] = mapped_column(Integer, nullable=True)
owner_id: Mapped[str | None] = mapped_column(String(36), nullable=True)
owner_generation: Mapped[str | None] = mapped_column(Text, nullable=True)
heartbeat_at: Mapped[datetime] = mapped_column(TZDateTime, default=utcnow)
started_at: Mapped[datetime] = mapped_column(TZDateTime, default=utcnow)
completed_at: Mapped[datetime | None] = mapped_column(TZDateTime, nullable=True)
Expand Down
Loading
Loading