fix(worker): address XBeg9's PR #39 follow-up findings - #67
Open
rcbevans wants to merge 1 commit into
Open
Conversation
8 fixes from XBeg9's final approved review (merged without being addressed — rcbevans: "To make progress I'll merge as is and address remaining comments suggestions in a follow up"): Worth fixing: 1. Cron loop error classification (leader.py): the cron loop hand-rolled its error classification with isinstance checks that missed 7 of 12 TRANSIENT_PG_ERRORS (DeadlockDetectedError, SerializationError, AdminShutdownError, CannotConnectNowError, TooManyConnectionsError, IdleSessionTimeoutError, IdleInTransactionSessionTimeoutError). Deadlock/serialization inside a transaction are routine, but 5 consecutive now killed the worker via the backstop guard. Fixed by catching TRANSIENT_PG_ERRORS first, keeping the narrower conn-state check inside it. Backlog: 2. Guard-before-cleanup ordering (leader.py): guard.unexpected(exc) raised before cleanup at all 4 backstop sites (election probe, lock attempt, upsert, watchdog probe). On the fatal iteration cleanup never ran, leaving is_leader set and leader_conn dangling. Fixed by moving cleanup before guard.unexpected at all 4 sites. 3. guard.ok() reset on probe path (leader.py): the probe's unexpected branch didn't continue, so the iteration fell through to re-election and reached guard.ok() at the bottom — a probe-only fault alternated unexpected/ok and never reached the cap. Fixed by adding continue. 4. Producer/validator mismatch (settings.py): the staleness invariant checked the producer loop, but the producer is not wrapped in asyncio.timeout (dispatch_batch is multi-statement). The timeout + period model doesn't hold. Removed the producer from the validator; updated the description. 5. connect() no timeout (deps.py): open_dedicated_conn passed command_timeout but not timeout= to asyncpg.connect, so connection establishment kept the driver's 60s default. Fixed by passing timeout=command_timeout. 6. Cron timeout continues past sleep (leader.py): the deadline and conn-state branches used continue, skipping the trailing asyncio.sleep(1). A persistently slow PG was re-issued BEGIN + tick_cron back-to-back. Fixed by removing continue so all branches fall through to the sleep. 7. settings.py wording: "every bounded loop" was inaccurate — 3 loops (leader.election, leader.watchdog, progress_flush) are not checked. Updated to say "the period-1 leader loops". 8. _transient.py doc: QueryCanceledError was described as "asyncpg's OTHER shape for a fired command_timeout". XBeg9 tested with real PG 18: command_timeout raises TimeoutError, not QueryCanceledError. QueryCanceledError is server-side 57014 (pg_cancel_backend or server-side statement_timeout). Fixed the comment. 14 new TDD regression tests in tests/test_pr39_followup_fixes.py. Updated test_settings.py to reflect the producer's removal from the validator. Verification: unit 4011 passed, integration 986 passed, e2e 52 passed, ruff clean, pyright clean (0 new errors).
XBeg9
self-requested a review
July 30, 2026 14:09
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
8 fixes from XBeg9's final approved review on #39 (merged without being addressed — rcbevans: "To make progress I'll merge as is and address remaining comments suggestions in a follow up").
Worth fixing
1. Cron loop error classification (
leader.py): the cron loop hand-rolled its error classification withisinstancechecks that missed 7 of 12TRANSIENT_PG_ERRORS(DeadlockDetectedError,SerializationError,AdminShutdownError,CannotConnectNowError,TooManyConnectionsError,IdleSessionTimeoutError,IdleInTransactionSessionTimeoutError). Deadlock/serialization inside a transaction are routine, but 5 consecutive now killed the worker via the backstop guard. Fixed by catchingTRANSIENT_PG_ERRORSfirst, keeping the narrower conn-state check inside it.Backlog
2. Guard-before-cleanup ordering (
leader.py):guard.unexpected(exc)raised before cleanup at all 4 backstop sites (election probe, lock attempt, upsert, watchdog probe). On the fatal iteration cleanup never ran, leavingis_leaderset andleader_conndangling. Fixed by moving cleanup beforeguard.unexpectedat all 4 sites.3.
guard.ok()reset on probe path (leader.py): the probe's unexpected branch didn'tcontinue, so the iteration fell through to re-election and reachedguard.ok()at the bottom — a probe-only fault alternated unexpected/ok and never reached the cap. Fixed by addingcontinue.4. Producer/validator mismatch (
settings.py): the staleness invariant checked the producer loop, but the producer is not wrapped inasyncio.timeout(dispatch_batchis multi-statement). Thetimeout + periodmodel doesn't hold. Removed the producer from the validator; updated the description.5.
connect()no timeout (deps.py):open_dedicated_connpassedcommand_timeoutbut nottimeout=toasyncpg.connect, so connection establishment kept the driver's 60s default. Fixed by passingtimeout=command_timeout.6. Cron timeout continues past sleep (
leader.py): the deadline and conn-state branches usedcontinue, skipping the trailingasyncio.sleep(1). A persistently slow PG was re-issuedBEGIN + tick_cronback-to-back. Fixed by removingcontinueso all branches fall through to the sleep.7. settings.py wording: "every bounded loop" was inaccurate — 3 loops (
leader.election,leader.watchdog,progress_flush) are not checked. Updated to say "the period-1 leader loops".8. _transient.py doc:
QueryCanceledErrorwas described as "asyncpg's OTHER shape for a firedcommand_timeout". XBeg9 tested with real PG 18:command_timeoutraisesTimeoutError, notQueryCanceledError.QueryCanceledErroris server-side 57014 (pg_cancel_backendor server-sidestatement_timeout). Fixed the comment.Testing
14 new TDD regression tests in
tests/test_pr39_followup_fixes.py— each written RED first, then GREEN. Updatedtests/test_settings.pyto reflect the producer's removal from the validator.Verification
test_property_sweep_equivalencepasses on rerun)ruff check,ruff format --check: cleanpyright: 0 new errors (pre-existing missing-extra errors only)