Repository navigation
fix(remote): wake up requests waiting on a server when the connection is lost - #394
Merged
Merged
Conversation
peshence
marked this pull request as ready for review
October 29, 2026 20:02
peshence
force-pushed
the
remote-connection-loss
branch
2 times, most recently
from
November 6, 2026 21:43
ae1feb1 to
663c741
Compare
danovaro
force-pushed
the
remote-connection-loss
branch
from
November 9, 2026 17:40
7ae2da2 to
f1791d6
Compare
…re is restarted The client neither reconnects nor resumes the request, so the reader stays blocked even though the store is back up. The test script can now write a script that restarts a server, and the harness shared by the remote store tests moves to a header.
… catalogue is restarted Same as for the store, for the requests waiting on RemoteFDB: the client neither reconnects nor resumes them, so they stay blocked even though the catalogue is back up.
… lost handleConnectionError() didn't call closeConnection(), so readers waiting on the store never got an error.
…nection is lost RemoteFDB did not override closeConnection(), so readers waiting on a list, axes, etc. never got an error when the connection to the catalogue was lost.
danovaro
force-pushed
the
remote-connection-loss
branch
from
November 10, 2026 07:34
f1791d6 to
834e919
Compare
This was referenced Nov 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
If the connection to the store or the catalogue is lost while a request is waiting on it, the caller waits forever instead of getting an error. I ran into this with a long-running client that kept hanging after the store dropped its connections.
When a listener thread dies,
handleConnectionError()fails the pending control requests and tears the connection down, but nothing wakes up readers waiting on a request queue (a retrieve throughRemoteStore, or results streaming throughRemoteFDB).closeConnection()does that, but it only runs on a cleanExit. The fix calls it fromhandleConnectionError()too, and givesRemoteFDBacloseConnection()of its own (it had none). Readers now get "Unexpected closure of store" or "Unexpected closure of remote FDB".The first two commits are the tests, one for the store and one for the catalogue, the last two are the fixes. Each test registers a pending request, kills the server, restarts it and expects the waiting reader to get an error. Without the fix the reader is still blocked 10s later: the client neither reconnects nor resumes the request, it just sits there.
test_server.sh.incan now write a script to restart a server, and the harness shared by the store tests moved to a header.Silent drops (packets just stop arriving, no reset) aren't solved by this alone. With the default Linux settings the client only notices after about 15 minutes with data in flight, or about 2 hours when idle. Before these fixes even that wouldn't end the hang; now it does. Shorter
net.ipv4.tcp_keepalive_*/tcp_retries2sysctls on the client bring it down to seconds (about 8s with 5s/2s/3 probes in my runs; the gist below has a compose repro for this,run-silent.sh). A better fix would be for fdb to set keepalive on its own sockets, with config knobs, so it doesn't depend on host settings. That's beyond the scope of this PR.I also reproduced both cases end to end in docker compose, with a proxy that resets the store's data connection and with the store going silent (
run-silent.sh): https://gist.github.com/peshence/8921b3c4a4019490f08d7bfc5b6da935🌈🌦️📖🚧 Documentation FDB 🚧📖🌦️🌈
https://sites.ecmwf.int/docs/fdb/pull-requests/PR-394