Repository navigation
Set the unpack temporary directory mode without the file server - #236
Merged
Merged
Conversation
make_tmp_dir/1 restricted the temporary directory with file:change_mode/2, which sends write_file_info to the file server. Unpacking is meant to make no file info requests to it, and unpack_bypasses_file_server_test has failed on main since #233 and #234 were both merged. The mode is now set with file:write_file_info/3 and the raw option, like the other file info calls when unpacking.
ericmj
added a commit
to hexpm/hex
that referenced
this pull request
Oct 6, 2026
This brings in hexpm/hex_core#232, hexpm/hex_core#233, hexpm/hex_core#234 and hexpm/hex_core#236: * Repository credentials are resolved without the :global lock unless an OAuth token has to be exchanged or refreshed. Every registry and tarball request took the lock, and requests that lost the race retried after random sleeps that grew exponentially. * Unpacking writes files and file metadata with raw file operations instead of through the file server process. * The temporary directory used while unpacking is created inside the output directory, or in the system temporary directory when there's no output directory, instead of in the current working directory. * Tar names that end partway through a UTF-8 sequence are no longer padded with zeros. mix deps.get for a project with the 100 Hex packages in hexpm's mix.lock went from 2.72 s to 2.15 s, the median of 15 interleaved runs with an empty Hex cache and no deps directory, on macOS against repo.hex.pm.
ericmj
added a commit
to hexpm/hex
that referenced
this pull request
Oct 6, 2026
* Update vendored hex_core to 4fbc5d1 This brings in hexpm/hex_core#232, hexpm/hex_core#233, hexpm/hex_core#234 and hexpm/hex_core#236: * Repository credentials are resolved without the :global lock unless an OAuth token has to be exchanged or refreshed. Every registry and tarball request took the lock, and requests that lost the race retried after random sleeps that grew exponentially. * Unpacking writes files and file metadata with raw file operations instead of through the file server process. * The temporary directory used while unpacking is created inside the output directory, or in the system temporary directory when there's no output directory, instead of in the current working directory. * Tar names that end partway through a UTF-8 sequence are no longer padded with zeros. mix deps.get for a project with the 100 Hex packages in hexpm's mix.lock went from 2.72 s to 2.15 s, the median of 15 interleaved runs with an empty Hex cache and no deps directory, on macOS against repo.hex.pm. * Unpack packages in parallel while they are fetched Mix calls Hex.SCM.update/1 for one dependency at a time, and each call unpacked the package from the cache into deps/<app>. For the 100 Hex packages in hexpm's mix.lock that was about 750 ms of serial unpacking after the downloads had finished. The fetch job started by Hex.SCM.prefetch/1 now also unpacks the package into deps/.<app>.hex-staging. update/1 verifies the checksums of the unpacked package against the registry, removes deps/<app> and renames the staging directory into its place. deps/<app> is still only changed by update/1, so Mix writes mix.lock after the same steps as before. If the job fails to unpack the package, update/1 unpacks it into the staging directory again to report the error. Since the checksums are verified before deps/<app> is removed, a package that fails to unpack or doesn't match the registry no longer removes the previously fetched dependency. This depends on the vendored hex_core creating its temporary unpack directory inside the output directory. With a relative temporary directory, Mix changing the working directory to load a rebar dependency while a job unpacked left temporary directories in other dependencies. mix deps.get for the 100 Hex packages in hexpm's mix.lock went from 2.10 s to 1.61 s, the median of 15 interleaved runs with an empty Hex cache and no deps directory, on macOS against repo.hex.pm. Peak RSS went from 222 MB to 227 MB.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
unpack_bypasses_file_server_testfails onmainsince #233 and #234 were both merged: #234'smake_tmp_dir/1restricts the temporary directory withfile:change_mode/2, which sendswrite_file_infoto the file server. The mode is now set withfile:write_file_info/3and therawoption, like the other file info calls when unpacking.