Repository navigation
Stop serializing concurrent fetches and unpacks on shared locks - #233
Merged
Merged
Conversation
resolve_repo_auth/2 ran under a global:trans/4 lock on
{hex_cli_auth, repo, RepoName} for every call, and resolving the global
OAuth token took a second lock, {hex_cli_auth, token_refresh}. with_repo/3
resolves credentials for every registry and tarball request, so a build
tool fetching packages concurrently queued all of its requests on these
locks. A process that finds a global lock taken retries after a random
sleep whose upper bound doubles with each attempt, so requests waited
far longer than the work done under the lock.
Credentials that need no exchange or refresh are now resolved without
the locks: a repo_key or auth_key from the auth config, a per-repository
or global OAuth token that has not expired, or no credentials. Exchanging
an auth_key and refreshing a global token still take their lock and
resolve again under it, so concurrent callers exchange or refresh once.
mix deps.get of a project with 100 packages from hex.pm, with this
hex_core vendored into Hex (median of 12 runs, no stored credentials):
2.83s to 2.42s at Hex's default 8 concurrent requests, and 2.55s to
1.90s at 32.
Unpacking to a directory set the mode and mtime of each extracted entry with file:write_file_info/3 in hex_erl_tar, and hex_tarball then reset the mtime of every file and directory with file:read_link_info/2 and file:write_file_info/3. Without the raw option each of these calls is a request to the file server, a single process per node, so tarballs unpacked in parallel processes were serialized on it. File contents were already written in raw mode. These calls now pass the raw option, as do the keep_old_files check, the writes of device and FIFO placeholder files, and the write of hex_metadata.config. Directories are still created through the file server, once per directory, since file:make_dir/1 has no raw option. Unpacking the 101 tarballs of a 100-package project 8 at a time went from 544ms to 365ms (median of 5 runs). One at a time, which is how Hex unpacks dependencies today, it went from 712ms to 697ms.
ericmj
force-pushed
the
ericmj/unpack-and-auth-perf
branch
from
October 5, 2026 18:01
935fda6 to
872f110
Compare
ericmj
added a commit
that referenced
this pull request
Oct 5, 2026
make_tmp_dir/1 restricted the temporary directory with file:change_mode/2, which sends write_file_info to the file server. Unpacking is meant to make no file info requests to it, and unpack_bypasses_file_server_test has failed on main since #233 and #234 were both merged. The mode is now set with file:write_file_info/3 and the raw option, like the other file info calls when unpacking.
ericmj
added a commit
to hexpm/hex
that referenced
this pull request
Oct 6, 2026
This brings in hexpm/hex_core#232, hexpm/hex_core#233, hexpm/hex_core#234 and hexpm/hex_core#236: * Repository credentials are resolved without the :global lock unless an OAuth token has to be exchanged or refreshed. Every registry and tarball request took the lock, and requests that lost the race retried after random sleeps that grew exponentially. * Unpacking writes files and file metadata with raw file operations instead of through the file server process. * The temporary directory used while unpacking is created inside the output directory, or in the system temporary directory when there's no output directory, instead of in the current working directory. * Tar names that end partway through a UTF-8 sequence are no longer padded with zeros. mix deps.get for a project with the 100 Hex packages in hexpm's mix.lock went from 2.72 s to 2.15 s, the median of 15 interleaved runs with an empty Hex cache and no deps directory, on macOS against repo.hex.pm.
ericmj
added a commit
to hexpm/hex
that referenced
this pull request
Oct 6, 2026
* Update vendored hex_core to 4fbc5d1 This brings in hexpm/hex_core#232, hexpm/hex_core#233, hexpm/hex_core#234 and hexpm/hex_core#236: * Repository credentials are resolved without the :global lock unless an OAuth token has to be exchanged or refreshed. Every registry and tarball request took the lock, and requests that lost the race retried after random sleeps that grew exponentially. * Unpacking writes files and file metadata with raw file operations instead of through the file server process. * The temporary directory used while unpacking is created inside the output directory, or in the system temporary directory when there's no output directory, instead of in the current working directory. * Tar names that end partway through a UTF-8 sequence are no longer padded with zeros. mix deps.get for a project with the 100 Hex packages in hexpm's mix.lock went from 2.72 s to 2.15 s, the median of 15 interleaved runs with an empty Hex cache and no deps directory, on macOS against repo.hex.pm. * Unpack packages in parallel while they are fetched Mix calls Hex.SCM.update/1 for one dependency at a time, and each call unpacked the package from the cache into deps/<app>. For the 100 Hex packages in hexpm's mix.lock that was about 750 ms of serial unpacking after the downloads had finished. The fetch job started by Hex.SCM.prefetch/1 now also unpacks the package into deps/.<app>.hex-staging. update/1 verifies the checksums of the unpacked package against the registry, removes deps/<app> and renames the staging directory into its place. deps/<app> is still only changed by update/1, so Mix writes mix.lock after the same steps as before. If the job fails to unpack the package, update/1 unpacks it into the staging directory again to report the error. Since the checksums are verified before deps/<app> is removed, a package that fails to unpack or doesn't match the registry no longer removes the previously fetched dependency. This depends on the vendored hex_core creating its temporary unpack directory inside the output directory. With a relative temporary directory, Mix changing the working directory to load a rebar dependency while a job unpacked left temporary directories in other dependencies. mix deps.get for the 100 Hex packages in hexpm's mix.lock went from 2.10 s to 1.61 s, the median of 15 interleaved runs with an empty Hex cache and no deps directory, on macOS against repo.hex.pm. Peak RSS went from 222 MB to 227 MB.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two places made concurrent work in a build tool run one at a time.
hex_cli_auth:resolve_repo_auth/2took aglobal:trans/4lock for every call, and the global OAuth token lookup took a second one.with_repo/3resolves credentials for every registry and tarball request, so concurrent fetches queued on these locks, andglobalretries a taken lock after a random sleep whose upper bound doubles each attempt. Credentials that need no exchange or refresh (a stored key, an unexpired token, or none) are now resolved without the locks. Exchanging anauth_keyand refreshing a global token still lock and resolve again under the lock. With this vendored into Hex,mix deps.getof a 100-package project went from 2.83s to 2.42s at Hex's default 8 concurrent requests and from 2.55s to 1.90s at 32 (median of 12 runs, no stored credentials).Unpacking to a directory set each entry's mode and mtime, and then reset every mtime, through the file server, a single process per node. File contents were already written in raw mode. Those file info calls now use
rawas well, so unpacks in parallel processes no longer wait on each other. Unpacking the 101 tarballs of that project 8 at a time went from 544ms to 365ms. Hex unpacks one dependency at a time today, where it changes 712ms to 697ms.