fix: rebuild async client when its loop-bound primitive is on a different loop (3.8.7) by palantir-valiot[bot] · Pull Request #81 · valiot/pygqlc

palantir-valiot · 2026-07-31T00:56:30Z

Description

async_execute now recovers when the shared httpx.AsyncClient's internal asyncio primitive (the httpcore connection-pool Event/Lock) is bound to a different event loop than the one it's being called from.

Root cause

The reported error (OPS-5447) was:

Error processing workflow QUEUE_REPLENISHMENT_FOR_CSV_REPORT:
<asyncio.locks.Event object at 0x7f218c298650 [unset]> is bound to a different event loop
  ... trigger_by_subscription.py:51  asyncio.run(maybe_awaitable)
  ... helper_subscriptions_trigger.py:88  workflow_config = await gql_activities.get_workflow_config(workflow_code)
  ... valiotworkflows/workflows/gql_activities.py:471  raise RuntimeError(str(QueryException(config_errors)))

I traced this through the installed stack rather than trusting the issue's guess at valiotworkflows:

valiotworkflows.gql_activities captures self._loop/self._executor in __init__ but never uses them — not the source. There is no module/class-level asyncio.Event anywhere in valiotworkflows or pygqlc.
get_workflow_config → self._gql.async_query → GraphQLClient.async_execute → self._async_client.post(...). The asyncio.Event is httpx/httpcore-internal, living on the long-lived GraphQLClient._async_client (a singleton).
That client is first bound to whichever loop touches it (a Temporal worker's loop). valuechainos_queues/services/workflows/trigger_by_subscription.py:51 creates a fresh loop per subscription callback via asyncio.run(), so client.post runs on a different loop → asyncio.mixins._LoopBoundMixin._get_loop() raises RuntimeError(f'{self!r} is bound to a different event loop').
pygqlc already recovers from the sibling "Event loop is closed" / "client has been closed" RuntimeErrors by dropping the stale client and rebuilding it on the current loop (async_execute's retry path). It simply did not recognize this message, so the error propagated into config_errors and was re-raised by get_workflow_config.

This is genuinely a cross-repo root cause: the issue is filed against valiot/ValueChainOS-Queues, but the fix must live in pygqlc — no change in ValueChainOS-Queues or valiotworkflows can fix it (the shared GraphQLClient is used from multiple loops). A persistent-loop change in trigger_by_subscription alone would not fix it either, because the same GraphQLClient is shared with the Temporal worker's loop.

Fix

_should_retry_on_fresh_connection now admits "is bound to a different event loop" alongside "Event loop is closed" / "client has been closed". The existing isinstance(e, RuntimeError) branch in async_execute then drops the stale client (_drop_async_client) and rebuilds it on the current loop (_get_async_client) before retrying — the same recovery path already used for a closed loop. One-line behavioral change; no new branches, no API change.

Consumers (valuechainos-queues depends on pygqlc>=3.8.6) pick this up on next install once 3.8.7 is published. A follow-up bump of that floor to >=3.8.7 is optional belt-and-suspenders.

Fixes OPS-5447

Type of change

Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

TDD — failing test first, then the one-line predicate change to green.

New parametrize case in test_should_retry_on_fresh_connection: the exact production RuntimeError("<asyncio.locks.Event object …> is bound to a different event loop") must return True (was False → red).
New test_async_execute_rebuilds_client_on_bound_to_different_event_loop: mirrors the existing closed-loop rebuild test — async_execute drops the stale client, rebuilds on the current loop, and the retry succeeds (was re-raising → red).
uv run pytest tests/pygqlc/gql_client/test_transient_transport_retry.py tests/pygqlc/gql_client/test_async_client_lifecycle.py tests/pygqlc/gql_client/test_timeout_empty_message.py → 25 passed.
uvx ruff format --check pygqlc/ tests/ → 34 files already formatted (changed files clean).
uvx bandit -r . -s B101 -ll --exclude $(find . -type d -name '.venv' | paste -sd, -) → No issues identified.
uv lock --upgrade + uv sync --frozen → lockfile committed (required by python-check-outdated.yaml).

Test Configuration:

Python 3.13.14 (pinned via mise.toml)
Toolchain: uv
SDK: pygqlc 3.8.7

Notes on the wider suite

uv run pytest (full) reports 25 passed + 25 errors. The 25 errors are all SystemExit: Check your environment variables from the session-scoped gql fixture in tests/conftest.py, which requires API/WSS/TOKEN secrets (CI injects them via ci.yml; they are unavailable in this sandbox). They are pre-existing and env-gated, and none touch the async-client retry path this PR changes. README.md has a pre-existing ruff format drift (unrelated, not a CI gate — ci.yml runs only pytest); left untouched to keep this diff scoped to the bug.

Checklist:

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes
Any dependent changes have been merged and published in downstream modules

💬 Continue the agent chat (live transcript, tools, follow-ups):
https://palantir.valiot.io/agent-tasks?task=04a5407f-fa44-44ff-b485-599aac487eb3

…rent loop (3.8.7) async_execute's _should_retry_on_fresh_connection now admits the RuntimeError '<asyncio.locks.Event object ...> is bound to a different event loop' alongside 'Event loop is closed' / 'client has been closed'. The shared GraphQLClient is a long-lived singleton, so its httpx.AsyncClient (and the connection-pool Event/Lock inside it) is first bound to whichever loop touches it (a Temporal worker's loop). When a caller later runs on a fresh loop — valuechainos_queues' trigger_by_subscription creates one per subscription callback via asyncio.run() — client.post raises that RuntimeError, surfacing as get_workflow_config failing every QUEUE_REPLENISHMENT_FOR_CSV_REPORT trigger (OPS-5447). async_execute drops the stale client and rebuilds it on the current loop before retrying — the same recovery path already used for a closed loop. uv.lock refreshed via uv lock --upgrade.

linear-code · 2026-07-31T00:56:33Z

OPS-5447

Copilot

Pull request overview

Note

Copilot could not run the full agentic suite for this review because it was automatically requested on a bot-authored pull request. Request a review from Copilot under Reviewers to retry with the full agentic suite. Improved support for bot-authored pull requests is coming soon.

Adds recovery for async_execute when the shared httpx.AsyncClient is tied to a different asyncio event loop, by treating that RuntimeError as retryable and rebuilding the client before retrying.

Changes:

Extend _should_retry_on_fresh_connection to treat “bound to a different event loop” as retryable.
Add an async test to ensure async_execute drops/rebuilds the async client on that failure mode.
Bump version and document the fix in the changelog.

Reviewed changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated 4 comments.

File	Description
tests/pygqlc/gql_client/test_transient_transport_retry.py	Adds regression coverage for rebuilding the async client when an asyncio primitive is bound to a different loop.
pygqlc/version.py	Bumps package version to 3.8.7 for the fix release.
pygqlc/GraphQLClient.py	Updates retry predicate to detect the “different event loop” RuntimeError and clarifies retry/rebuild behavior.
CHANGELOG.md	Documents the new recovery behavior and its real-world trigger.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+            # Only a dead/wrong event loop needs a full rebuild — and those are
+            # the only RuntimeErrors the predicate admits. Dropping the shared
+            # pool per transient transport error would churn connections.


+        if (
+            "Event loop is closed" in msg
+            or "client has been closed" in msg
+            or "is bound to a different event loop" in msg
+        ):


+        # An asyncio primitive (httpx/httpcore's connection-pool Event/Lock) is
+        # bound to the loop the shared client was first used on; calling from a
+        # fresh loop (asyncio.run per callback, a Temporal loop vs. a listener
+        # loop) raises this — a fresh client on the current loop is the fix.
+        (
+            RuntimeError(
+                "<asyncio.locks.Event object at 0x7f218c298650 [unset]> is bound to a different event loop"
+            ),
+            True,
+        ),


+        side_effect=RuntimeError(
+            "<asyncio.locks.Event object at 0x7f218c298650 [unset]> "
+            "is bound to a different event loop"
+        )
+    )


palantir-valiot · 2026-07-31T01:41:14Z

✅ Closing as redundant

The base branch advanced and this fix is no longer needed — A competing fix (PR #82, commit 45fb003) already merged to main as 3.8.7 with the same root-cause resolution: it adds the identical 'is bound to a different event loop' clause to _should_retry_on_fresh_connection (origin/main:GraphQLClient.py:1176) AND a superior per-loop httpx.AsyncClient cache (weak-keyed by running loop) that prevents cross-loop binding outright. main is already at version 3.8.7 with a CHANGELOG entry citing the same QUEUE_REPLENISHMENT_FOR_CSV_REPORT error. My PR's core change is a strict subset of what landed; the remaining diff is only duplicate version/CHANGELOG entries. The bug no longer reproduces on the new base.

Closing this PR automatically. If that's wrong, reopen it and leave a
comment explaining what's still broken.

🤖 Implementer · GLM 5.2 (Fireworks)

Copilot AI review requested due to automatic review settings July 31, 2026 00:56

palantir-valiot Bot requested a review from acrogenesis July 31, 2026 00:56

Copilot AI reviewed Jul 31, 2026

View reviewed changes

palantir-valiot Bot added palantir:working palantir:queued labels Jul 31, 2026

palantir-valiot Bot closed this Jul 31, 2026

palantir-valiot Bot removed palantir:queued palantir:working labels Jul 31, 2026

Navigation Menu

Uh oh!

fix: rebuild async client when its loop-bound primitive is on a different loop (3.8.7) - #81

fix: rebuild async client when its loop-bound primitive is on a different loop (3.8.7)#81
palantir-valiot[bot] wants to merge 1 commit into
mainfrom
palantir/OPS-5447-cross-loop-async-client-recovery

palantir-valiot Bot commented Jul 31, 2026 •

edited

Loading

Uh oh!

linear-code Bot commented Jul 31, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

palantir-valiot Bot commented Jul 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

palantir-valiot Bot commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Root cause

Fix

Type of change

How Has This Been Tested?

Notes on the wider suite

Checklist:

Uh oh!

linear-code Bot commented Jul 31, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

palantir-valiot Bot commented Jul 31, 2026

✅ Closing as redundant

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

palantir-valiot Bot commented Jul 31, 2026 •

edited

Loading