Repository navigation
Set libgit2 server timeouts to prevent hung git fetches - #996
Merged
Merged
Conversation
A stalled SSH fetch via Rugged holds the GVL indefinitely, freezing all Ruby threads including the rufus scheduler. Confirmed in production: the process hung mid-fetch with no further output from any thread, including the $stdout.write-based heartbeat, until the pod was restarted. libgit2 1.7+ supports GIT_OPT_SET_SERVER_CONNECT_TIMEOUT and GIT_OPT_SET_SERVER_TIMEOUT but Rugged::Settings does not yet expose them. Call git_libgit2_opts directly via Fiddle at boot to set both to 5 minutes. Enum values verified against libgit2 1.9.6 (bundled in rugged 1.9.6); confirm against the bundled version if upgrading. Once a Rugged PR lands to add these options to rugged_settings.c, the Fiddle workaround can be replaced with: Rugged::Settings["server_connect_timeout"] = timeout_ms Rugged::Settings["server_timeout"] = timeout_ms
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A stalled SSH fetch via Rugged holds the GVL indefinitely, freezing all Ruby threads including the rufus scheduler. Confirmed in production: the process hung mid-fetch with no further output from any thread, including the $stdout.write-based heartbeat, until the pod was restarted.
libgit2 1.7+ supports GIT_OPT_SET_SERVER_CONNECT_TIMEOUT and GIT_OPT_SET_SERVER_TIMEOUT but Rugged::Settings does not yet expose them. Call git_libgit2_opts directly via Fiddle at boot to set both to 5 minutes. Enum values verified against libgit2 1.9.6 (bundled in rugged 1.9.6); confirm against the bundled version if upgrading.
Once a Rugged PR lands to add these options to rugged_settings.c, the Fiddle workaround can be replaced with:
@bdunne Please review. I've created a Rugged PR at libgit2/rugged#1015 but it's waiting on review, and I'd like to get this temp fix in to see if it even works.