Repository navigation
Drop the zero padding of tar names that end in incomplete UTF-8 - #232
Merged
Merged
Conversation
hex_erl_tar's parse_string/1 reads a header field up to the first NUL and decodes it as UTF-8. When a name ended partway through a UTF-8 sequence, it fell back to the bytes of the whole field instead of the part before the NUL, so the name kept the zero padding. A package whose inner tarball has the ustar name "café" in ISO Latin-1 unpacked to a 100 character name: "caf", 233 and 96 zeros. The fallback now converts only the bytes before the NUL. The same fix is proposed for OTP in erlang/otp#11764.
ericmj
added a commit
to hexpm/hex
that referenced
this pull request
Oct 6, 2026
This brings in hexpm/hex_core#232, hexpm/hex_core#233, hexpm/hex_core#234 and hexpm/hex_core#236: * Repository credentials are resolved without the :global lock unless an OAuth token has to be exchanged or refreshed. Every registry and tarball request took the lock, and requests that lost the race retried after random sleeps that grew exponentially. * Unpacking writes files and file metadata with raw file operations instead of through the file server process. * The temporary directory used while unpacking is created inside the output directory, or in the system temporary directory when there's no output directory, instead of in the current working directory. * Tar names that end partway through a UTF-8 sequence are no longer padded with zeros. mix deps.get for a project with the 100 Hex packages in hexpm's mix.lock went from 2.72 s to 2.15 s, the median of 15 interleaved runs with an empty Hex cache and no deps directory, on macOS against repo.hex.pm.
ericmj
added a commit
to hexpm/hex
that referenced
this pull request
Oct 6, 2026
* Update vendored hex_core to 4fbc5d1 This brings in hexpm/hex_core#232, hexpm/hex_core#233, hexpm/hex_core#234 and hexpm/hex_core#236: * Repository credentials are resolved without the :global lock unless an OAuth token has to be exchanged or refreshed. Every registry and tarball request took the lock, and requests that lost the race retried after random sleeps that grew exponentially. * Unpacking writes files and file metadata with raw file operations instead of through the file server process. * The temporary directory used while unpacking is created inside the output directory, or in the system temporary directory when there's no output directory, instead of in the current working directory. * Tar names that end partway through a UTF-8 sequence are no longer padded with zeros. mix deps.get for a project with the 100 Hex packages in hexpm's mix.lock went from 2.72 s to 2.15 s, the median of 15 interleaved runs with an empty Hex cache and no deps directory, on macOS against repo.hex.pm. * Unpack packages in parallel while they are fetched Mix calls Hex.SCM.update/1 for one dependency at a time, and each call unpacked the package from the cache into deps/<app>. For the 100 Hex packages in hexpm's mix.lock that was about 750 ms of serial unpacking after the downloads had finished. The fetch job started by Hex.SCM.prefetch/1 now also unpacks the package into deps/.<app>.hex-staging. update/1 verifies the checksums of the unpacked package against the registry, removes deps/<app> and renames the staging directory into its place. deps/<app> is still only changed by update/1, so Mix writes mix.lock after the same steps as before. If the job fails to unpack the package, update/1 unpacks it into the staging directory again to report the error. Since the checksums are verified before deps/<app> is removed, a package that fails to unpack or doesn't match the registry no longer removes the previously fetched dependency. This depends on the vendored hex_core creating its temporary unpack directory inside the output directory. With a relative temporary directory, Mix changing the working directory to load a rebar dependency while a job unpacked left temporary directories in other dependencies. mix deps.get for the 100 Hex packages in hexpm's mix.lock went from 2.10 s to 1.61 s, the median of 15 interleaved runs with an empty Hex cache and no deps directory, on macOS against repo.hex.pm. Peak RSS went from 222 MB to 227 MB.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
hex_erl_tar's
parse_string/1reads a header field up to the first NUL and decodes it as UTF-8. When a name ends partway through a UTF-8 sequence, the{incomplete, ...}branch falls back tobinary_to_list/1on the whole field instead of the part before the NUL, so the name keeps the field's zero padding. A package whose inner tarball has the ustar namecaféin ISO Latin-1 (last byte 16#E9) unpacked to a 100 character name:"caf", 233 and 96 zeros. hex_tarball never writes such names, since non-ASCII names go into PAX headers, so this only affects tarballs built by other tools.The fallback now converts only the bytes before the NUL, listed as change 9 in the hex_erl_tar header. The same fix is up for OTP in erlang/otp#11764.
The new test builds a package, rewrites the inner tarball's first ustar name to those bytes, recomputes
CHECKSUMand checks the namehex_tarball:unpack/2returns.