Skip to content

fix(worker): address XBeg9's PR #39 follow-up findings - #67

Open
rcbevans wants to merge 1 commit into
mainfrom
fix/pr39-followup-fixes
Open

fix(worker): address XBeg9's PR #39 follow-up findings#67
rcbevans wants to merge 1 commit into
mainfrom
fix/pr39-followup-fixes

Conversation

@rcbevans

Copy link
Copy Markdown
Contributor

What

8 fixes from XBeg9's final approved review on #39 (merged without being addressed — rcbevans: "To make progress I'll merge as is and address remaining comments suggestions in a follow up").

Worth fixing

1. Cron loop error classification (leader.py): the cron loop hand-rolled its error classification with isinstance checks that missed 7 of 12 TRANSIENT_PG_ERRORS (DeadlockDetectedError, SerializationError, AdminShutdownError, CannotConnectNowError, TooManyConnectionsError, IdleSessionTimeoutError, IdleInTransactionSessionTimeoutError). Deadlock/serialization inside a transaction are routine, but 5 consecutive now killed the worker via the backstop guard. Fixed by catching TRANSIENT_PG_ERRORS first, keeping the narrower conn-state check inside it.

Backlog

2. Guard-before-cleanup ordering (leader.py): guard.unexpected(exc) raised before cleanup at all 4 backstop sites (election probe, lock attempt, upsert, watchdog probe). On the fatal iteration cleanup never ran, leaving is_leader set and leader_conn dangling. Fixed by moving cleanup before guard.unexpected at all 4 sites.

3. guard.ok() reset on probe path (leader.py): the probe's unexpected branch didn't continue, so the iteration fell through to re-election and reached guard.ok() at the bottom — a probe-only fault alternated unexpected/ok and never reached the cap. Fixed by adding continue.

4. Producer/validator mismatch (settings.py): the staleness invariant checked the producer loop, but the producer is not wrapped in asyncio.timeout (dispatch_batch is multi-statement). The timeout + period model doesn't hold. Removed the producer from the validator; updated the description.

5. connect() no timeout (deps.py): open_dedicated_conn passed command_timeout but not timeout= to asyncpg.connect, so connection establishment kept the driver's 60s default. Fixed by passing timeout=command_timeout.

6. Cron timeout continues past sleep (leader.py): the deadline and conn-state branches used continue, skipping the trailing asyncio.sleep(1). A persistently slow PG was re-issued BEGIN + tick_cron back-to-back. Fixed by removing continue so all branches fall through to the sleep.

7. settings.py wording: "every bounded loop" was inaccurate — 3 loops (leader.election, leader.watchdog, progress_flush) are not checked. Updated to say "the period-1 leader loops".

8. _transient.py doc: QueryCanceledError was described as "asyncpg's OTHER shape for a fired command_timeout". XBeg9 tested with real PG 18: command_timeout raises TimeoutError, not QueryCanceledError. QueryCanceledError is server-side 57014 (pg_cancel_backend or server-side statement_timeout). Fixed the comment.

Testing

14 new TDD regression tests in tests/test_pr39_followup_fixes.py — each written RED first, then GREEN. Updated tests/test_settings.py to reflect the producer's removal from the validator.

Verification

  • unit: 4011 passed, 9 skipped
  • integration: 986 passed, 6 skipped (1 flaky test_property_sweep_equivalence passes on rerun)
  • e2e: 52 passed (all 41 original + 4 admin/SSE + 7 others)
  • ruff check, ruff format --check: clean
  • pyright: 0 new errors (pre-existing missing-extra errors only)

8 fixes from XBeg9's final approved review (merged without being
addressed — rcbevans: "To make progress I'll merge as is and address
remaining comments suggestions in a follow up"):

Worth fixing:
1. Cron loop error classification (leader.py): the cron loop hand-rolled
   its error classification with isinstance checks that missed 7 of 12
   TRANSIENT_PG_ERRORS (DeadlockDetectedError, SerializationError,
   AdminShutdownError, CannotConnectNowError, TooManyConnectionsError,
   IdleSessionTimeoutError, IdleInTransactionSessionTimeoutError).
   Deadlock/serialization inside a transaction are routine, but 5
   consecutive now killed the worker via the backstop guard. Fixed by
   catching TRANSIENT_PG_ERRORS first, keeping the narrower conn-state
   check inside it.

Backlog:
2. Guard-before-cleanup ordering (leader.py): guard.unexpected(exc)
   raised before cleanup at all 4 backstop sites (election probe, lock
   attempt, upsert, watchdog probe). On the fatal iteration cleanup
   never ran, leaving is_leader set and leader_conn dangling. Fixed by
   moving cleanup before guard.unexpected at all 4 sites.

3. guard.ok() reset on probe path (leader.py): the probe's unexpected
   branch didn't continue, so the iteration fell through to re-election
   and reached guard.ok() at the bottom — a probe-only fault alternated
   unexpected/ok and never reached the cap. Fixed by adding continue.

4. Producer/validator mismatch (settings.py): the staleness invariant
   checked the producer loop, but the producer is not wrapped in
   asyncio.timeout (dispatch_batch is multi-statement). The
   timeout + period model doesn't hold. Removed the producer from the
   validator; updated the description.

5. connect() no timeout (deps.py): open_dedicated_conn passed
   command_timeout but not timeout= to asyncpg.connect, so connection
   establishment kept the driver's 60s default. Fixed by passing
   timeout=command_timeout.

6. Cron timeout continues past sleep (leader.py): the deadline and
   conn-state branches used continue, skipping the trailing
   asyncio.sleep(1). A persistently slow PG was re-issued BEGIN +
   tick_cron back-to-back. Fixed by removing continue so all branches
   fall through to the sleep.

7. settings.py wording: "every bounded loop" was inaccurate — 3 loops
   (leader.election, leader.watchdog, progress_flush) are not checked.
   Updated to say "the period-1 leader loops".

8. _transient.py doc: QueryCanceledError was described as "asyncpg's
   OTHER shape for a fired command_timeout". XBeg9 tested with real PG
   18: command_timeout raises TimeoutError, not QueryCanceledError.
   QueryCanceledError is server-side 57014 (pg_cancel_backend or
   server-side statement_timeout). Fixed the comment.

14 new TDD regression tests in tests/test_pr39_followup_fixes.py.
Updated test_settings.py to reflect the producer's removal from the
validator.

Verification: unit 4011 passed, integration 986 passed, e2e 52 passed,
ruff clean, pyright clean (0 new errors).
@XBeg9
XBeg9 self-requested a review July 30, 2026 14:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant