feat(io): native XET reads for hf:// models via xet-core C-API - #1
Draft
XciD wants to merge 10 commits into
Draft
Conversation
Read HuggingFace files through the XET content-addressed storage (CAS) protocol instead of the resolve/ byte-range path in hf.zig. - xet_core.zig: CAS read-token exchange, Session (CAS client + reconstructor) with readRange / downloadToWriter, and listXetFiles. The protocol itself (chunk reconstruction, LZ4/BG4, BLAKE3) is delegated to the jedisct1/zig-xet package. Everything is driven by std.Io. - xet.zig: repo-scoped VFS provider (Xet), mirroring http.zig handle bookkeeping; fileReadPositional reconstructs only the requested byte range. - Register Xet / xet_core in index.zig, wire the @xet dep in BUILD.bazel, and document the remaining Bazel repo declaration in XET_INTEGRATION.md.
Pivot the XET download support from a from-scratch Zig reimplementation to driving the mature xet-core (Rust) client through its C-API (hf_xet.h). This matches ZML's own parallel HTTP downloader at line rate (~1 GB/s vs ~386 MB/s for a Zig reimplementation) and gets BLAKE3 verification + the on-disk chunk cache for free. - xet_hub.zig: pure-Zig HF Hub helpers (CAS read-token exchange, file listing). - xet_capi.zig: Zig binding over hf_xet.h (@cImport); Session + downloadToPath. - xet.zig: high-level downloadFile() (download-to-cache model, like hf-xet). - Drop the zig-xet-based xet_core.zig and the @xet zig package dep; BUILD.bazel now links libxet_capi (cc_import). See XET_INTEGRATION.md for the build recipe and the benchmark numbers. Verified: compiled with Zig 0.16.0 + libxet_capi (xet-core assaf/c-api branch); xet.downloadFile ran end-to-end against huggingface.co (full 1.16 GB file).
XciD
changed the title
Cleanups from a review pass, no behavior change: - xet_hub: factor the GET+bearer+parse pipeline into one getJson helper, share a single caller-owned std.http.Client across both HF calls (was one per call), and use the typed authorization request field instead of a manual extra header. Validate JSON value types before use. - xet_hub: ReadToken now owns NUL-terminated strings, dropping the extra dupeZ the C-API call previously needed. - xet.zig: drop dead re-exports, match the file path exactly (the endsWith fallback could mis-select), and pass the token straight to Session.init. - xet_capi: drop the download-report take that was only freeing the report. - Document the PoC limitations (not a VFS provider, per-file session, whole-tree metadata fetch, blocking poll) in XET_INTEGRATION.md.
hf.zig performRead now detects XET-backed files on first access and reconstructs the whole file once into a local cache via the xet-core C-API (download-to-cache), then serves reads from the local file; non-XET files keep the existing resolve range-GET path. So hf:// XET files go through xet-core transparently. xet_capi.zig now declares the C ABI with `extern` decls instead of @cImport, so the build only links libxet_capi.a (no header include path to thread through Bazel). Verified in ZML's real Bazel build (//examples/io:playground, Zig 0.16.0 + libxet_capi from xet-core assaf/c-api): the integrated `cp hf://` reconstructs a full 4.98 GB file at ~228 MB/s vs ~23 MB/s for the lazy resolve path (~10x). See XET_INTEGRATION.md. Note: examples/io cp aborts on exit in ZML's own file:// provider close path, unrelated to this change.
Instead of eagerly downloading the whole file to a cache on first read, hf.zig now reconstructs each requested byte range on demand via the xet-core C-API stream API (readRange with has_range). This keeps ZML's lazy positional-read model: a partial read fetches only the covering xorbs, and overlapping ranges hit the chunk cache. Non-XET files keep the resolve range-GET path. xet_capi.zig binds the download-stream range API (Session.readRange); xet.zig adds openRemote -> RemoteFile for reusable per-file sessions, alongside the eager downloadFile. Measured on //examples/io:playground (EC2, 4.98 GB file), sequential cp: lazy ~36 MB/s, eager download-to-cache ~228 MB/s, resolve baseline ~23 MB/s, raw parallel xet ~1070 MB/s. Lazy pays per-read reconstruction on a sequential full-file copy; its win is partial/random reads. See XET_INTEGRATION.md.
VFS.lookupDir only treated a scheme (hf://...) as absolute when the base dir was cwd. Opening a scheme-qualified path relative to an already-open dir of the same backend (as safetensors.fromPath does: open the model dir, then open the full hf:// file path under it) double-prefixed the dir's path, so hf.zig's tree lookup returned FileNotFound. Now any "scheme://" sub_path resolves from the scheme's root regardless of the base dir. This unblocks loading hf:// XET models end-to-end. Measured on //examples/io:playground (EC2, real weight load of a 4.63 GiB shard): XET lazy reads ~348 MB/s vs ~158 MB/s for the resolve path (HF_XET_DISABLE=1), ~2.2x. Added the HF_XET_DISABLE escape hatch in hf.zig. See XET_INTEGRATION.md.
Cleanups from a review pass (no behavior change to the lazy read path): - Drop the unused eager download path (downloadFile / Session.downloadToPath / the file-download-group C-API surface). hf.zig only ever uses lazy range reads, so the file-download group was dead weight and one CAS group per session was always wasted. - hf.zig openXet now reuses HF's pooled http.Client and its already-formatted authorization header instead of spinning up a throwaway client and recovering the raw token by slicing "Bearer " off the header. - xet_hub: fold listXetFiles into findXetFile, which resolves one path and dupes only the matching hash instead of materializing and freeing the whole tree. - Drop the HF_XET_DISABLE debug gate (std.c.getenv in the read path) and unused public aliases. Verified: builds in ZML's Bazel build (//examples/io:playground) and the lazy readRange path reconstructs correctly.
XciD
changed the title
The reconstruction/read path only decompresses fetched chunks; it does not recompute or check chunk BLAKE3 hashes. BLAKE3 verification in xet-core is for xorb validation (upload/explicit) and tests, not the download hot path.
Resolving an hf:// XET file went through findXetFile, which fetched the full repo tree (up to 8MB) and linear-scanned it for a single file, plus requestReadToken, once per file. For a multi-shard model loaded in parallel this burst of identical tree requests rate-limited the Hub API into too_many_requests, so every file fell back to the resolve path and XET was never actually exercised. Fetch the repo tree once (indexed by path) and the read token once, cached on HF keyed by "repo_id@rev" and guarded by the existing mutex. openXet now does an O(1) lookup and builds the session from the cached hash/size/token, turning N tree fetches + N tokens into 1 + 1.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Wires the xet-core C-API (huggingface/xet-core#881) into ZML so
hf://XET-backed files reconstruct through xet-core, instead of the plainresolverange-GET path. Reads go through the C-API on demand, with a fallback toresolvefor non-XET files.Results
Real model weight load (a 4.63 GiB Llama-3.1-8B shard, parallel positional reads through
zml.io.TensorStore), m6i.2xlarge, same binary.HF_XET_DISABLE=1selects the baseline:About 2.8x. Raw parallel download tops out around 1 GB/s (the box's network, same as ZML's own downloader); the load figure is lower because it is real positional reads rather than a bulk copy.
How
xet_hub.zig: HF Hub HTTPS/JSON (CAS read-token exchange, resolve a file's XET hash + size).xet_capi.zig:externbinding over thehf_xet.hC ABI, aSessionwithreadRange. Symbols are declared inline, so the build only linkslibxet_capi.a(no header include path).xet.zig:openRemotereturning aRemoteFilewithreadRange.hf.zig:performReadreconstructs the requested range through the C-API for XET-backed files (reusing HF's pooled client and auth), and falls back toresolveotherwise.Dependencies and notes
libxet_capi.a, cross-compiled per target) wired into Bazel.safetensors.fromPathonhf://); without it, models fail to load.