feat(capi): add xet_capi C ABI crate for hf-xet - #881
Draft
assafvayner wants to merge 50 commits into
Draft
Conversation
…add stream task_id
Previously xet_op_take_* freed the op internally on success/error but not on the wrong-variant path, an inconsistent contract that made double-frees easy (freeing an already-consumed op joins an already-joined worker thread and panics with ESRCH). Switch to a single predictable rule: take_* never frees; the caller always frees every op exactly once with xet_op_free. This is the RAII/defer-friendly model the language bindings want.
Each performs the same upload -> commit -> download-by-hash round-trip against a real HF Xet repo via the C API, using a Hub token-refresh URL for auth. All four were tested end-to-end against assafvayner/xet-c-api-test.
Add a dedicated build_and_test-capi job that: - runs the xet_capi FFI test suite with the simulation feature (header freshness, smoke.c compile against the header, and an end-to-end upload/commit/download round-trip over a local CAS, no network), and - links tests/smoke.c against the built cdylib and runs it, exercising real ABI linkage (the Rust c_smoke_compiles test only compiles the TU). Add a main() to smoke.c so it is directly runnable in CI. Assisted-by: Claude <noreply@anthropic.com>
Move xet_session.hpp from examples/cpp/ to include/ so the header-only C++ wrapper ships alongside hf_xet.h as part of the public C API surface, and update the C++ example Makefile to reference the new location. Add xet_capi/CMakeLists.txt exposing an imported xet_capi target (built via cargo) that carries the include directory and the platform native-link libraries, so consumers can add_subdirectory() and link it directly. Assisted-by: Claude <noreply@anthropic.com>
…apper Fill in missing doc comments across xet_capi's public FFI items (free functions, XetStatus/XetSha256Policy/XetPollState variants, XetHeader/ XetAuthConfig fields) so the generated hf_xet.h is fully documented, and add a Doxyfile to render it as HTML. Move xet_session.hpp out of include/ (C-only) into examples/cpp/ alongside its only consumer. Note in the examples README that the C API is experimental.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
xet_capi, a new workspace crate exposing the fullxet::xet_session::XetSessionsurface over a C ABI, sohf-xetcan be consumed from C / C++ / CGo. It follows the same thin-wrapper pattern as the existing Python (hf_xet), wasm (hf_xet_wasm), and Node (xet_pkg_napi) bindings.Design: opaque handles freed explicitly; async transfers via handle-based polling (
XetOp—poll+ typedtake_*), backed by std threads running the existing_blockingAPIs; no callbacks cross the ABI in either direction. Errors areXetStatuscodes plus an opaqueXetErrorout-param. Reports are opaque with accessor functions; scalar progress/dedup are flat#[repr(C)]structs. The headerinclude/hf_xet.his cbindgen-generated and committed.Surface
xet_session_new/free,xet_init_logging, and threeXetAuthConfig-driven builders (upload commit, file-download group, stream group).XetFileInfo, download-to-path, finish, progress, abort, task_id; ordered/unordered streaming with pull-basednext, progress, cancel, task_id.XetOppoll + typedtake_*(file-metadata / commit-report / download-report / bytes / chunk / void / error).XetProgress/XetDedupMetrics;XetBytes;XetError/XetStatus.Build & safety
crate-type = ["cdylib", "staticlib", "rlib"]; release build produceslibxet_capi.{a,dylib}.extern "C"fn is wrapped in a panic guard (ffi_guard/catch_unwind) so panics never unwind across the ABI; all pointer args null-checked.build.rs; the committed header is guarded by an up-to-date test and a symbol-presence test.Test Plan
cargo test -p xet_capi— 15 tests pass (default features)cargo test -p xet_capi --features simulation— 16 pass, includinge2e_upload_then_download_via_ffi(real upload → download round-trip through the C ABI over alocal://CAS)cargo clippy -p xet_capi --all-targets -- -D warningsand--features simulation— cleancargo +nightly fmt -p xet_capi --check— cleancargo build -p xet_capi --release— produces staticlib + cdylibc_smoke_compiles—tests/smoke.ccompiles against the generated headerinclude/hf_xet.hagainst intended C consumer usage