Skip to content

fix(remote): wake up requests waiting on a server when the connection is lost - #394

Merged
danovaro merged 4 commits into
developfrom
remote-connection-loss
Nov 10, 2026
Merged

danovaro merged 4 commits into
developfrom
remote-connection-loss

Conversation

@peshence

@peshence peshence commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

If the connection to the store or the catalogue is lost while a request is waiting on it, the caller waits forever instead of getting an error. I ran into this with a long-running client that kept hanging after the store dropped its connections.

When a listener thread dies, handleConnectionError() fails the pending control requests and tears the connection down, but nothing wakes up readers waiting on a request queue (a retrieve through RemoteStore, or results streaming through RemoteFDB). closeConnection() does that, but it only runs on a clean Exit. The fix calls it from handleConnectionError() too, and gives RemoteFDB a closeConnection() of its own (it had none). Readers now get "Unexpected closure of store" or "Unexpected closure of remote FDB".

The first two commits are the tests, one for the store and one for the catalogue, the last two are the fixes. Each test registers a pending request, kills the server, restarts it and expects the waiting reader to get an error. Without the fix the reader is still blocked 10s later: the client neither reconnects nor resumes the request, it just sits there. test_server.sh.in can now write a script to restart a server, and the harness shared by the store tests moved to a header.

Silent drops (packets just stop arriving, no reset) aren't solved by this alone. With the default Linux settings the client only notices after about 15 minutes with data in flight, or about 2 hours when idle. Before these fixes even that wouldn't end the hang; now it does. Shorter net.ipv4.tcp_keepalive_* / tcp_retries2 sysctls on the client bring it down to seconds (about 8s with 5s/2s/3 probes in my runs; the gist below has a compose repro for this, run-silent.sh). A better fix would be for fdb to set keepalive on its own sockets, with config knobs, so it doesn't depend on host settings. That's beyond the scope of this PR.

I also reproduced both cases end to end in docker compose, with a proxy that resets the store's data connection and with the store going silent (run-silent.sh): https://gist.github.com/peshence/8921b3c4a4019490f08d7bfc5b6da935

🌈🌦️📖🚧 Documentation FDB 🚧📖🌦️🌈
https://sites.ecmwf.int/docs/fdb/pull-requests/PR-394

@peshence
peshence marked this pull request as ready for review October 29, 2026 20:02
@peshence peshence changed the title test(remote): a request waiting on the store should fail when the connection is lost fix(remote): don't hang forever when the connection to the store is lost" Oct 30, 2026
@peshence
peshence force-pushed the remote-connection-loss branch 2 times, most recently from ae1feb1 to 663c741 Compare November 6, 2026 21:43
@peshence peshence changed the title fix(remote): don't hang forever when the connection to the store is lost" fix(remote): wake up requests waiting on a server when the connection is lost Nov 6, 2026
@danovaro
danovaro force-pushed the remote-connection-loss branch from 7ae2da2 to f1791d6 Compare November 9, 2026 17:40
@danovaro danovaro added downstream-ci-not-needed Deliberately skip the downstream fan-out run-hpc-ci Run the HPC legs of CI labels Nov 9, 2026
…re is restarted

The client neither reconnects nor resumes the request, so the reader stays
blocked even though the store is back up.

The test script can now write a script that restarts a server, and the harness
shared by the remote store tests moves to a header.
… catalogue is restarted

Same as for the store, for the requests waiting on RemoteFDB: the client
neither reconnects nor resumes them, so they stay blocked even though the
catalogue is back up.
… lost

handleConnectionError() didn't call closeConnection(), so readers waiting on
the store never got an error.
…nection is lost

RemoteFDB did not override closeConnection(), so readers waiting on a list,
axes, etc. never got an error when the connection to the catalogue was lost.
@danovaro
danovaro force-pushed the remote-connection-loss branch from f1791d6 to 834e919 Compare November 10, 2026 07:34
@danovaro
danovaro merged commit ad6c0d6 into develop Nov 10, 2026
39 checks passed
@danovaro
danovaro deleted the remote-connection-loss branch November 10, 2026 07:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

downstream-ci-not-needed Deliberately skip the downstream fan-out run-hpc-ci Run the HPC legs of CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants