Repository navigation
[Engine] Inkling-2-Mini from the official NVFP4/MXFP8 checkpoint, expert offload and CPU experts for 8-16 GB cards - #6520
baptisterajaut wants to merge 73 commits into
Conversation
…-vram-reserve-mib)
…perts, key-split attention)
…ify and draft, adaptive fallback, --spec in --serve
…ack after 3 iterations
…ls, effort words, stop token, Strata kv names)
…xpert memory, background promotion into VRAM
…ns, per-node doorbells so pools touch local memory only
…ed first, admission margin) instead of LRU
…GLU, warm AVX-512, pinned engine thread
…sured CPU and PCIe times
…f page cache no longer spills it across UPI
…ate_up (resident decode +4%, bit-identical)
… arguments, README, drop the pre-server shim
# Conflicts: # terminal/serve/frontend.py
…t attention's smem attribute only without --tc --kv k8v4: K as in --kv i8 (int8, fp16 scale per 64), V through the Walsh-Hadamard rotation of --kv i8r and stored as 4-bit codes with an fp16 scale per 32 values; the attention output is rotated back. 1632 bytes per cell per slot (i8: 2112, f16: 4096), so the expert pools get the rest. Turing (sm_75) has 64 KiB of shared memory per block: the fp32 prompt attention (kb_attn, 83 KiB) cannot run there and its attribute call failed at startup even with --tc 1, which does not use it. The attribute is set only without --tc, with a message when the card cannot.
…p (k_attn_partial 23.8 -> 13.6 us, decode back to 92 tok/s)
K as in --kv i8, V through the Walsh-Hadamard rotation of --kv i8r and stored int8 (fp16 scale per 64); q is not rotated, the attention output is rotated back. Same bytes as i8. With routing forced to an f32-K/V run (Inkling-2-Mini safetensors, 1338 positions): mean KL 1.24e-3, p99 6.8e-3, top-1 99.48% (i8: 1.36e-3, 1.9e-2, 98.43%; i8r: 2.19e-3, 1.7e-2, 99.25%).
…ols get the rest The fixed 384 MiB beside --vram-reserve-mib covered cuBLAS' lazy workspace and the decode graph; cuBLAS now runs its shapes at load (before the pools are sized), so 128 MiB is enough. RTX 2060 SUPER, IQ3_S GGUF, --kv i8, chunk 4096, reserve 64: 1426 -> 1497 expert slots, decode 30.1 -> 31.1 tok/s (23-token prompt), 24.4 -> 25.3 tok/s (5269-token prompt).
…red at 8 GiB (17-31 MiB used after the pools)
…IQ3_S / IQ4_NL / Q8_0 experts --model FILE.gguf reads llama.cpp's merged layout (#32694, b12600+): - dense matrices in Q8_0, repacked losslessly into int8 codes + fp16 scales, with their own GEMV (decode) and fp32 / fp16 conversions (prompt); - the router (F32, exact) and the loop gates as BF16; - the Q8_0 embedding table in host RAM; - experts gate_up / down in IQ3_S, IQ4_NL or Q8_0: one slot per expert [gate_up rows | down rows] in pool_nv (every layer one format, no MXFP8 pool), read through the part A / part B tables like the checkpoint formats, so the offload (device LRU, prompt streaming) works unchanged. Their decode GEMV and prompt grouped products (SIMT and tensor-core) decode the ggml blocks on the fly. The converter's transforms are undone at load: static conv taps (flip(0).T), rel_logits_proj (transposed), gate_up rows (gate then up, read through a row permutation into the interleaved gu layout). The F32 tensors match the safetensors bit for bit after that.
…there), short chunks skip the NVFP4/MXFP8 small-T paths
--kv-grow 1: the full-attention slots' K/V and (with offload) the expert pools live in CUDA virtual address ranges. The K/V is mapped in --kv-grow-step cells (8192) as the context grows; when VRAM is short the pools give up slots (k_pool_evict drops their experts from the tables, the decode graph is rebuilt for the new slot count). A shorter new request hands the cells back and the pools refill. The addresses never move.
…ns and the spec verify graphs (slot counts baked in)
…er and oracle venv notes
… by the picture hash (vis_id), the image bypass and vis_reuse gone
…maps to the effort none
…uild, tokenizer pack, card-sized engine args)
…serve leave the engine its minimum expert pools)
… beside the engine minimum pools
…annot configure them with nvcc 13)
…iB peak in offload) instead of Flash-Next 34-55 GB
… that keeps the engine minimum expert pools (k8v8 / chunk 256 on 8 GB cards)
…s shrink their chunk, a prompt that cannot fit ends with ERR context full instead of exiting
… K/V to f32, measured) where f16 does not reach 32K
…onfig keeps mapped at the floor (730 MiB, measured)
…he from the RAM left (4096 / 2048 MiB, off with a reason)
|
Out of draft. Since it was opened, the conversation cache, pictures, the setup family and a docs page landed, and every number in the description was re-run on the final engine (2daf740; the head only adds The conversation cache keeps conversations in pinned RAM with a checkpoint at each prompt end, because agent clients send the history back without the reasoning and the server re-renders it, so prefix reuse never fired after the first turn. With two agents taking turns, the returning turns wait 8.6 s in all for their first token instead of 22.3 s resident, and 36-37 s instead of 100 s at 8 GB. Pictures run the checkpoint's ViT in the engine in fp32, at mean KL 1e-7 or less against transformers, and take no VRAM before the first picture. A 960x686 picture gives its first token in 0.63 s resident and 2.27 s at 8 GB.
Two corrections to the draft. The 8 GB figure of 65.7-67.0 tok/s does not hold: the build that gave it now measures 62 with the same flags, so the gap comes from the machine, and setup's config is the 66 above. Thinking-off requests were broken on Inkling, because its template ignores Tool calls were checked with BFCL v4 multi_turn_base: 1,199 calls, all parsed with valid arguments, none left in the answer text. The score of that 100-task subset is in the description with its caveats. |
|
Original NVFP4 weights Implemented via Marlin or how? |
|
Blackwell numbers for this PR, since the description says nothing here was run on one: RTX 5060 Ti 16 GB (sm_120), the official NVFP4 + MXFP8 checkpoint, expert offload with CPU experts. It builds, runs and answers correctly; numbers below. One finding about GGUF loading at the end. Setup
What the engine chose
Numbers (two separate starts, 15:30 and 15:57; the same within noise)
So about 945 tok/s of prompt at 4.8K and about 990 tok/s at 15-18K, decode 55-57 tok/s with 19 % of the experts off the card. Five short Japanese questions (capital, 17 × 23, a proverb, a Python dedupe function, photosynthesis in three sentences) all reached a correct final answer (105-518 tokens, thinking included: with For reference, llama.cpp on the same machine (b12627 release build, CUDA 13.5,
(llama.cpp's numbers are from its own GGUF: Not tested: Tested with help from Claude Code (Anthropic), which ran the builds and measurements; I checked the results. |
Nope, own kernels. And @Norhiro thanks for the numbers and for the report of the ggml failure. I'll look at it. Please also note your prefill is limited here due to no proper Blackwell support (I don't have one) so prefill should go faster |
I rewrote the Marlin kernels from SGLang for this and they're significantly faster. According to my Codex benchmark report, I'm getting 52 tok/s without DSpark, compared to 26 tok/s with your kernels, also without DSpark. So roughly a 2× speedup. |
… NUMA offload, and Dual-GPU EP Unify official Hugging Face safetensors (thinkingmachines/Inkling-2-Mini + mtp.safetensors + vision) and community/Unsloth GGUF checkpoints (UD-Q4_K_XL, IQ3_S, IQ4_NL, IQ4_XS, Q4_K, Q5_K, Q6_K, Q8_0, MXFP4, NVFP4, split multi-shard, and external dspark GGUF/safetensors) directly inside strata-inkling while keeping native .strata v1.1 (SQ-20/15/10 + Vision) as the flagship recommended default. - Zero-requantization official HF safetensors adapter (ink_safetensors_loader.h): maps NVFP4 L0..19 straight to SF4 (code_id=15), MXFP8 L20..23 straight to MX8 (code_id=16), BF16 dense trunk, 3x loop-weighted vram_load_order, mtp.safetensors DSpark drafter, and model.visual.* vision weights (ink_vision.h). - Community / Unsloth GGUF adapter (ink_gguf_loader.h): supports both llama.cpp master (GGUF_shard_rewrite) and PR #32187 layouts, split multi-shard GGUFs, bit-exact NVFP4 -> SF4 nibble-desplit repack, lossless SkCode::GGUF_RAW (id=20) native block dispatch across CPU and CUDA, full-array 200K o200k tokenizer + chat_template export, and Unsloth dspark-*.gguf (dflash arch) 54-tensor mapping. - Ported PR #6520 CUDA-Graph mapped-memory doorbell + NUMA-aware AVX-512/AVX2 CPU expert worker pools + decayed-use background promotion (hybrid_doorbell.{h,cu}) and --kv k8v8|k8v4 KV compression (ink_batch.h, hybrid_doorbell.cu). - Ported PR #6506 Dual-GPU Expert Parallelism (--gpus 2 / --ep-split / IK2_EP_SPLIT) with P2P / pinned fallback and native SF4/MX8/S_K/GGUF_RAW GPU kernels (ep_dual_gpu.{h,cu}) in modular non-conflicting files. - Wired multi-format detection, community HF sources, local import args, and HF o200k tokenizer loading into terminal/setup.py, terminal/strata_cli/launch/catalog.py, terminal/tools/strata_tokenizer.py, and terminal/serve/test_inkling_formats.py. - Preserved original MIT copyright headers (Project Maya, Baptiste Rajaut, merbanan, cobra91) and documented contributor credits for 0.2.19 release notes.
[Engine] inkling2: AMD ROCm/HIP support for strata-ink2, strata-ink2ref, and setup
…ild; docs name AMD RDNA
Problem and result
Strata cannot run Inkling-2-Mini (thinkingmachines/Inkling-2-Mini: 128-expert top-8 MoE, layers 6-17 looped three times, so 48 layer applications per token, plus a ViT for pictures). This adds
strata-ink2, an engine for that family that loads the official checkpoint as shipped (NVFP4 experts for layers 0-19, MXFP8 for 20-23, BF16 dense weights and vision tower), installs it from./setup.sh --family inkling2, and serves it from the Strata server with its chat template, thinking and tool-call markers, reasoning-effort words, pictures and a conversation cache.This duplicates work from #6506 (cobra91), but I went the other path on purpose: the official checkpoint is already quantized enough to work on as is, so there is no requantization, and the quality is the reference's. Against the transformers implementation (huggingface/transformers#51050) in fp32, the token path with fp32 K/V is at mean KL 1.4e-12 on a 1,600-token text, top-1 identical on every position; #6506 and llama.cpp both run a Q4 GGUF at KL 9e-4 on the same kind of text (measured in my comment on #6506).
The PR is big because the goal was not a minimal engine strapped on in a hurry, but an integration that is already good where Strata matters, on 8-16 GB cards. On an RTX 3090 limited to 8 GB, with the experts that do not fit computed on the CPU, it decodes at 60-66 tok/s with a 32K context.
Related discussion
#6442, opened 2026-11-20 before any code. A public WIP branch was posted there at 11:05 UTC on 2026-11-21, more than three hours before #6506 was opened. Benchmarks of #6506 and maya against this branch are in my comments on #6506.
Approach
strata-ink2is a standalone CUDA engine (engine/src/inkling2/). Decode is one captured graph per token. Prefill uses cuBLAS for the dense products and a tiled grouped GEMM for the experts, with weights decoded on the fly.strata-ink2refis the plain reference path used to check it, andterminal/tools/inkling2/has the oracle and KL tools.--vram-reserve-mib Nleaves N MiB of VRAM free, and the experts that do not fit live in pinned RAM. Placement uses decayed use counts with an admission margin, and promotions run in the background.--cpu-threads Ncomputes the non-resident experts on the CPU: AVX-512 or AVX2 kernels, one thread pool per NUMA node, each expert's memory bound to its node. Doorbells in mapped memory, written from inside the decode graph, tell the pools what to compute, so there is no host sync per layer.--kv-grow 1(merbanan's) lets the K/V grow into the expert pools as a conversation grows. Each pool keeps 64 slots: a prompt shrinks its chunk near that floor, and one that cannot fit ends with a "context is full" error while the engine keeps serving.--conversation-cache-mib Nkeeps conversations in pinned RAM. A checkpoint at each prompt end lets the next turn continue even when the client sends the history back without the model's reasoning (most agent clients do, and the server re-renders the history, so plain prefix reuse never fired after turn 1). Conversations set aside, an agent and its subagents taking turns, are taken up again, and a long shared beginning is read once.image=keys). The conversation cache keys picture tokens on the picture's content.--spec 1uses the model's own DSpark drafter (mtp.safetensors). Greedy decoding stays exact, sampled decoding keeps the target's distribution, and it falls back to plain decode when that is faster.--spec: GGUF loading (IQ3_S / IQ4_NL / Q8_0),--kv k8v4/k8v8,--kv-grow, elastic prompt buffers, routed short prompts, a decode-once GGUF prompt GEMM, a chunk size that shrinks to fit, a smaller offload margin, and a Turing startup fix.inkling2family with its own installer: the checkpoint at a pinned revision with every file's size and SHA-256, the tokenizer pack, thestrata-ink2build, and engine arguments sized from the card, the RAM and what a desktop already holds. Per card size, and what each config measures, is indocs/INKLING2.md.Everything engine-side is opt-in or specific to
strata-ink2/strata-ink2ref. Outside them: the server's Inkling family switch and image gate, a fix for thinking-off requests on Inkling (enable_thinking: falseused to leave effort 0.9 in the prompt), and setup's CMake minimum raised to 3.25.2, because 3.24 and Debian 12's 3.25.1 cannot configure the engine's CUDA20 targets with nvcc 13; that last one is a separate commit and can be split out.Validation (RTX 3090 24 GB, 2x Xeon Gold 6252, CUDA 13.2, Linux; engine and server at 2daf740; the later commits add the docs page and the AMD port, whose CUDA build has the same SASS and PTX and gives bit-identical outputs; the picture and conversation-cache identity checks ran on their branches before integration)
--specgives the same 128 tokens as plain decode, resident and at 8 GB.--spec), prompt 2,380 tok/s.--spec,--kv-grow, 8 GB with offload on the GPU only). With two agents taking turns and their reasoning dropped from the history (6 requests, prompts from 6K to 17K tokens), the four returning turns wait 8.6 s in all for their first token instead of 22.3 s resident, and 36-37 s instead of 100 s at 8 GB with a desktop../setup.sh, the server passes its smoke checks on the 24 GB card and a simulated 8 GB one (thinking on/off, effort words, a tool call round trip, streaming, a 30K-token prompt, a picture).python3 -m pytest serveandtests/terminalinterminal/: no new failures (see below).test_stop_closed_pipe.pyandtest_mcp_image_default.py, which fail the same way on main at 6e38f58 on this machine; and 23 tests plus 24 collection errors intests/terminal(notextualin the system Python), the same set as on this branch before the setup commits (48febba), not checked on main.--specexact, the server end to end. The targets are built on Linux HIP only and stay out of the default HIP build, so the other engines' HIP builds compile nothing new. SYCL: nothing new compiled.Limits
--cpu-threadson Linux only, at most two NUMA nodes.--cpu-threads, greedy output is not repeatable run to run: which misses go over PCIe depends on timings, and two runs can differ at mean KL about 1e-4.INK2_PCIE=0makes it repeatable.usage.prompt_tokens; the engine expands it (432 tokens for a 960x686 picture).--specand--cpu-threadsare not combined yet, so offload configs run without the drafter.--cpu-threadsand--spec, and setup does not install them.strata launchor the desktop app.Next, not in this PR
mma.syncexpert GEMM that decodes NVFP4 in registers, and an opt-in W4A8 path with its KL. A 4096-token chunk already reads prompts 23-24% faster than 1024 at a 32K context, but its buffers do not fit at 64K beside the drafter, so setup keeps 1024.--specwith--cpu-threads, for the drafter on small cards.--cpu-threads.