Add optional Hugging Face .tok tokenizer backend

[JacobSzwejbka]

Summary

• add an opt-in Hugging Face .tok inference backend behind TOKENIZERS_BUILD_HF_RUST_TOKENIZER=OFF
• expose it through the existing C++ tokenizers::Tokenizer interface as tokenizers::hf_rust_tokenizer
• keep JSON parsing, serialization, training, normalizers, and other heavy dependencies out of the runtime by linking only the pinned tk-encode and tk-serialization crates
• install and export the optional C++ and Rust archives for downstream consumers such as ExecuTorch

The format and parser-free runtime come from Hugging Face Tokenizers PR #2352 and the associated feat/tok-format work. A follow-up ExecuTorch change will advance this submodule and opt into the target only when requested.

Binary-size behavior

The option is disabled by default. In that configuration CMake does not enter the Rust directory, look for Cargo, compile the bridge, or add either archive, so the default tokenizer binary has a 0-byte delta.

Measured opt-in Apple arm64 artifacts with the size profile:

Artifact	Stripped	gzip -9
.tok C++/Rust bridge	591,040 B	298,961 B

Validation

• cargo fmt --check
• cargo clippy --locked --tests -- -D warnings
• cargo test --locked (3 tests)
• default-off CMake configure/build with Cargo absent from PATH
• C++17 adapter compilation with -Os -fno-exceptions -fno-rtti -Wall -Werror
• GPT-2 tokenizer.json converted to .tok, then loaded for encode, token lookup, and incremental decode
• installed CMake package consumed through find_package(tokenizers CONFIG REQUIRED)

Current scope

This draft supports host CMake builds. Android, Apple framework, WASM, and Buck packaging still require explicit Rust target/toolchain integration.