Repository navigation
Conversation
- Add Inkling2 to ModelFamily enum and GGUF architecture reader ('inkling2')
- Add Inkling 2 engine in engine/src/ik2 with NVFP4, FP8 KV attention and CUDA 13.x compatibility
- Add Inkling 2 family switch in strata main (generate.cpp) and --families support
- Register strata_ik2 library and strata-ik2 target in CMake
- Support Windows MSVC, Blackwell SM_120 and dual-GPU EP execution
… to mmap on Windows
…ly separate reasoning and content
…data in model table
|
Hi @cobra91. A few things about this PR, from someone who has no say in what gets merged here but has been working on the same model in #6442. The engine in this PR is What showed that the engine was correct was dropped in the copy: On the issues: a WIP branch with numbers was posted in #6442 more than three hours before this PR was opened. I don't find this a fair way to treat maya's authors or this repo's contributors. Crediting them and restoring their notice would seem the minimum. Once our own work is done, we will build this PR and maya's head on an RTX 3090 and post numbers here: KL against a reference, and decode speed all-resident and with offload. |
Hi @baptisterajaut, Thank you for the candid feedback. I completely understand why it looked that way from the outside, but there is a major piece of context you might have missed: I am actually the author of the Windows MSVC, CUDA 13.x, and Dual-GPU EP implementation in Project Maya. Having spent hours implementing and testing the Windows and dual-GPU architecture for Maya earlier today, our goal here was simply to bridge that very same work into Strata so our community could benefit from it (including the CLI, control server, and Desktop UI). That being said, you are 100% right on the licensing and attribution standard: That was poor practice, and I take full responsibility for not putting that front and center from commit 1. Here is what we are pushing right now:
Regarding #6442: we genuinely had not seen your branch before opening the PR. Since we both care about bringing Inkling 2 to Strata, there is zero reason to work in silos. We'd love to have your validation on your RTX 3090, incorporate your findings, or collaborate directly to make this PR as solid as possible. Pushing the updates now! |
|
I checked the updated Calling the actual Bounded CPU fixture, exact source hash and output. Command: Could this text-only case be included while finishing the existing setup correction? No competing PR is needed. |
|
@cobra91 I've been running on Turing and have a set of patches here: https://github.com/baptisterajaut/Strata/pulls Could you maybe pickup support for other quants in your PR? |
|
Numbers as promised, from one RTX 3090 24 GB. This PR at 176c365 and maya's #1095 head at 4d176d28099a, both on unsloth's Inkling-2-Mini-UD-Q4_K_XL.gguf (sha256 e62d463c…4de15: NVFP4 experts for layers 0-19, Q8_0 for the rest); ours is the feat/inkling2 branch on the official NVFP4 + MXFP8 safetensors. Same 257-token chat prompt, 512 greedy tokens, two runs each. All resident, this PR decodes at 120-124 tok/s and maya's head at the same speed, with bit-identical logits. With maya's drafter it reaches 130-134 tok/s, but its output leaves plain greedy at token 46, and the two runs differ from each other. Ours does 87.5 tok/s, so maya's engine is about 1.4x faster on that path, and that gap is ours to close. With the VRAM limited, each ik2 cache was sized so that peak VRAM matches ours (7.8 and 11.9 GiB). At 8 GB this PR does 22-24 tok/s, and maya's head 30 with IK2_DENSE=q8; ours does 63. At 12 GB: 45-47, 54, and 78. On a 256-token text, the KL against an fp32 reference (which matches HF transformers at ~1e-12) is 9e-4 mean for this PR and maya, top-1 253/256. llama.cpp on the same GGUF gives 9.9e-4, so that is the quant, not the engine. Broken in this PR: Scripts and logs are available if anyone wants to rerun them. |
… and --vram-reserve-mib
|
Thank you everyone for the valuable reviews, fixtures, and benchmark numbers! Here is the update pushed in commit
|
|
Nice update! Bit late (I was looking for this but couldn't find anything so started work on my own for polystrata, but could help here) A few findings from getting it working and tuned on a 8gb 4060 ti: Exact generation with --draft on looped models (n_pass = 3) Each of the 48 visits has its own 4 causal conv states (K, V, attn_post, ffn_post). By saving per-token conv state snapshots during the 8-token verification window (rolling back to the last accepted token's snapshot on a partial rejection) and overwriting the temporary anchor/mask K/V slots in the draft cache with the accepted tokens' context K/V (k_proj_ctx / v_proj_ctx), greedy generation stays 100% bit-identical to 1-token decode. dspark_norm.weight vs output_norm.weight in dflash GGUFs In the dflash GGUF, the two norm names are swapped relative to what you'd expect: output_norm.weight is the drafter's final norm (before the LM head), while dspark_norm.weight is the markov feature norm (markov_head.dflare_norm). Using them the other way around drops acceptance from ~3.69 tok/verify down to ~1.57 tok/verify. Folding dspark_hidden_out into the LM head at load time In the 7-step Markov loop, d_out[i+1] + Wh @ latent_i + b_h feeds straight into the LM head (output) before the softcap. Precomputing W_fold = quant_q4_0(output @ Wh) ([200058, 256]) and b_fold = output @ b_h once at load time lets us run base = output @ d_out[1..7] as a single n = 7 batched GEMM outside the loop, and shrinks the 7 sequential Markov GEMVs by 9× (from [200058, 2304] to [200058, 256]). 8 GB VRAM & DDR4 tuning: |
… Core speculative decoding
|
Huge thanks @Algo-Enthusiast for sharing these valuable findings and confirming the ~3.64 accepted tokens/step on Inkling 2 Mini (looped MoE, 48 visits)! We just pushed commit 1. Dual-GPU Resident
|
|
Not quite merged yet, I just opened a PR with my Windows changes 😅 Also included a bugfix for |
…sident CUDA graph fast-path
…ice pool sizing - P2P tail-skip via Ly.ep_has_peer: eliminate 100% of P2P transfers & sync events on core loop passes (75% of token passes) - Core resident GPU0 placement: pin layers 6..17 (1,536 experts) to GPU 0 (1,000 GB/s GDDR7), outer layers to GPU 1 - Per-device pool capacity capping: prevent Pool 0 from over-allocating on GPU 1 and starving Pool 1, achieving 100% VRAM residency - Enables zero-wait resident EP dispatch with CUDA Graph capture & instantiation - Benchmarked: 104.24 tok/s decode on RTX 5080 + 4060 Ti with Inkling-2-Mini-UD-Q4_K_XL (0 PCIe misses) - Tested prototype originally explored in lab/p2p-turbo Co-authored-by: kuraneko1 <https://github.com/kuraneko1>
🚀 perf(ik2): Asymmetric Dual-GPU EP with Core P2P Tail-Skip & Per-Device Pool SizingFollowing our previous progress, we just pushed commit 📌 Overview & Root CauseOn heterogeneous multi-GPU systems (such as RTX 5080 16GB @ 1,000 GB/s + RTX 4060 Ti 16GB @ 288 GB/s), a naive symmetric 50/50 expert split caps overall decoding throughput to the memory bandwidth of the slowest GPU (80.7 tok/s), while forcing P2P transfer and event synchronization across all 48 layer visits per token. Furthermore, we identified a critical pool sizing bug: the default allocator split pool slots proportionally to layer count across the whole model (e.g. 20 layers in Pool 0, 4 layers in Pool 1), causing a secondary GPU to over-allocate Pool 0 for experts it never owns while starving Pool 1, inadvertently causing unnecessary host RAM evictions. 🧠 What Commit
|
| Architecture / Execution Path | Prefill Speed | Decode Speed | Memory / Bus Status |
|---|---|---|---|
| Naive Symmetric Dual-GPU (50/50 split) | ~20.0 tok/s | 80.70 tok/s | P2P stalls on all 48 passes; limited by 4060 Ti 288 GB/s bus |
| Mono-GPU RTX 5080 (CoreResident hybrid) | 28.40 tok/s | 89.24 tok/s | 1,762 PCIe misses (exceeds single GPU 16GB VRAM) |
| Asymmetric Dual-GPU EP + Tail-Skip (This Commit) | 49.30 tok/s | 104.24 tok/s | 100% in VRAM (0 misses) + CUDA Graphs instantiated |
Net Gain: +29.2% decode speed over symmetric dual-GPU, +16.8% over mono-GPU, and breaking the 100 tok/s barrier on a 17.6 GB MoE model on consumer hardware.
🤝 Community Verification Request
If anyone in the thread (e.g. @baptisterajaut, @Algo-Enthusiast, @mrexodia, or others with 24GB cards such as RTX 3090 / 4090 or dual-3090 NVLink setups) has a chance to pull and test this commit, confirmation of decode speed and bit-exact validation on your end would be greatly appreciated!
Credits: Co-authored with @kuraneko1 for core-resident loop heuristics and hardware profiling insights; tail-skip architecture prototyped in lab/p2p-turbo.
… NUMA offload, and Dual-GPU EP Unify official Hugging Face safetensors (thinkingmachines/Inkling-2-Mini + mtp.safetensors + vision) and community/Unsloth GGUF checkpoints (UD-Q4_K_XL, IQ3_S, IQ4_NL, IQ4_XS, Q4_K, Q5_K, Q6_K, Q8_0, MXFP4, NVFP4, split multi-shard, and external dspark GGUF/safetensors) directly inside strata-inkling while keeping native .strata v1.1 (SQ-20/15/10 + Vision) as the flagship recommended default. - Zero-requantization official HF safetensors adapter (ink_safetensors_loader.h): maps NVFP4 L0..19 straight to SF4 (code_id=15), MXFP8 L20..23 straight to MX8 (code_id=16), BF16 dense trunk, 3x loop-weighted vram_load_order, mtp.safetensors DSpark drafter, and model.visual.* vision weights (ink_vision.h). - Community / Unsloth GGUF adapter (ink_gguf_loader.h): supports both llama.cpp master (GGUF_shard_rewrite) and PR #32187 layouts, split multi-shard GGUFs, bit-exact NVFP4 -> SF4 nibble-desplit repack, lossless SkCode::GGUF_RAW (id=20) native block dispatch across CPU and CUDA, full-array 200K o200k tokenizer + chat_template export, and Unsloth dspark-*.gguf (dflash arch) 54-tensor mapping. - Ported PR #6520 CUDA-Graph mapped-memory doorbell + NUMA-aware AVX-512/AVX2 CPU expert worker pools + decayed-use background promotion (hybrid_doorbell.{h,cu}) and --kv k8v8|k8v4 KV compression (ink_batch.h, hybrid_doorbell.cu). - Ported PR #6506 Dual-GPU Expert Parallelism (--gpus 2 / --ep-split / IK2_EP_SPLIT) with P2P / pinned fallback and native SF4/MX8/S_K/GGUF_RAW GPU kernels (ep_dual_gpu.{h,cu}) in modular non-conflicting files. - Wired multi-format detection, community HF sources, local import args, and HF o200k tokenizer loading into terminal/setup.py, terminal/strata_cli/launch/catalog.py, terminal/tools/strata_tokenizer.py, and terminal/serve/test_inkling_formats.py. - Preserved original MIT copyright headers (Project Maya, Baptiste Rajaut, merbanan, cobra91) and documented contributor credits for 0.2.19 release notes.
This is with 52 patches ontop of https://github.com/cobra91/Strata-DualGPU/tree/feat/inkling2-support. I think this is close to the max this GPU is capable of. |
|
Huge thanks @merbanan for testing our branch and sharing these impressive numbers! Getting ~53–58 tok/s decode on a consumer 8 GB card (RTX 2060 SUPER) with IQ3_S and int8 K/V is an extraordinary milestone. It confirms that with proper quant / KV support, this looped architecture can fly on consumer hardware. We'd love to see your 52 patches land alongside this PR! @Niko1221 (Niko Veit) We have cleanly rebased and merged To support integration into 📊 Benchmark Comparison: Consumer Hardware
Why this matters for consumer hardware:
Ready for review whenever you have a moment for |


Problem and result:
Strata currently supports Qwen architectures (
qwen4exp,qwen35moe), but has no native support for Thinking Machines Lab's Inkling 2 architecture (inkling2, 26.4B physical / ~5.4B active looped MoE with NVFP4 / Q8_0 weights and o200k tokenization).This PR adds complete end-to-end support for the Inkling 2 model family in Strata across the engine, control, CLI, and desktop UI.
Upstream Lineage & Individual Contributions:
Huge thanks to everyone across both Project Maya and Strata whose work and feedback made this integration possible:
ik2engine architecture and kernel scaffolding.ik2engine completion, NVFP4/MMA kernels, and speculative decoding.IK2_EP=1/IK2_EP_SPLIT) originally contributed in alesha-pro/project-maya#3 and merged into Maya PR kv_q8: the gather decodes 8 int8 per thread (V100 experimental build, +11%) #1095; full Strata integration across CLI, native engine bridge, Desktop UI, chat server, and o200k tokenizer.KeyError: mmprojandKeyError: mtp).--prefill autochunking and--vram-reserve-mibhandling.Licensing & Attribution:
engine/src/ik2/files and test suites:Copyright (c) 2026 Peasant Smith, OscarSheng, alesha-pro, cobra91 and the Project Maya contributors.ik2_iq_tables.h.What this PR includes:
engine/src/ik2/supporting looped MoE execution (12-layer core repeated across passes), sliding-window attention, relative bias, sinks, L2 clipping, and FP8/FP16 KV cache.--gpus 2,IK2_EP=1): Zero-copy mapped staging buffer (g_ep_stage, 648 KB) and multi-GPU resident CUDA Graph replay enabling ~80.7 tok/s decode on asymmetric PCIe dual-GPU rigs.--draft): Dynamic looped drafter support (4 taps / 3 layers for Mini, 8 taps for Small), multi-token Tensor Core MMA verify windows (ep_combine_block) achieving 62.8 tok/s with zero cache misses.KeyError: mmprojby moving projector path resolution inside the vision guard interminal/setup.py."mtp": 0entry toOWN_PATH_FAMILIES["inkling"].--prefill autogracefully inik2_engine.cu(defaulting chunk to 2048 instead of 0).--vram-reserve-mibintoik2_engine.cuto respect user-specified VRAM headroom reservations.engine/tests/cuda/ik2_kernels_test.cu), Python forward-pass reference (ref/inkling2.py), comparison utility (tools/ik2_compare.py), and documentation (docs/INKLING2.md).setup.pyforInkling-2-Mini-UD-Q4_K_XL.gguf, with explicitdrafter: nonesetting.strata_tokenizersupport for o200k (inkling2) pre-tokenization withignore_merges, plusInklingRewriter/InklingOutputParserinterminal/serve/frontend.pyto seamlessly parse Inkling's typed segments (<|content_thinking|>,<|content_text|>) into the Strata web chat UI, Desktop App, and/v1/chat/completionsAPI.Validation & Benchmark Performance:
sm_89,sm_120), and GCC/Clang on Linux (sm_86).mv_q8p,attn_res_rms_f16,experts_mma_kernel) eliminating the per-layer CPU launch overhead across the 48 looped visits.fast_token): 80.7 tok/s (up from 49.2 tok/s baseline) via dual-GPU resident CUDA graph and 648 KB zero-copy mapped EP staging with 0 cache misses.--draft): 62.8 tok/s (3.62 accepted tokens/step on code, 54.2 ms verification across 48 looped visits, 0 cache misses) accelerated by multi-token Tensor Core MMA (to_f16_limbs_tokens,ep_combine_block).