Repository navigation
Description
Area
[Engine] (speed, kernels, models, the server and API)
Desktop app, Terminal or source?
Source (START-HERE.bat, setup.sh or a build)
What happened
With the same model files, the same pack and the same expert profile, 0.2.15 decodes 28–36% slower than a 0.2.7-based build on an RTX 3060 12 GB. The decode expert-cache hit rate falls from ~96% to ~75–81%. The clearest signal is with adaptation off (--adapt-swaps 0): the resident set that --expert-cache auto builds from the profile hits 25–42% of routed experts on 0.2.15, against 72–81% on 0.2.7. Swapping then has to repair a much worse starting set, and it only gets part of the way.
I have not bisected it yet, so I don't know which commit changed this. I can bisect if that helps.
Measurements
1 agent, IQ3_S, one GPU, 131K context, --kv int8, MTP drafting on. "decode" = two fixed 400-token greedy chat answers. "new-prompt decode" = two 300-token answers on topics that are new in every run. Speed = (completion tokens − 1) / (last token time − first token time). Hit rate and PCIe share come from the server's decode expert cache hit rate lines. 1 run per row.
System A (Xeon E5-1650 v2 Ivy Bridge, AVX only; 64 GB DDR3-1600; RTX 3060 12 GB at PCIe 3.0 x16, capped at 115 W; 5 pool workers)
| Engine | Setting | Decode tok/s | New-prompt decode | Cache hit | Routed over PCIe |
|---|---|---|---|---|---|
| 0.2.7-based | default | 55.6 | 52.6 | 95.7–96.2% | 0.8–1.0% |
| 0.2.7-based | --adapt-swaps 0 |
44.8 | 36.8 | 75.0–81.3% | 5.0–6.6% |
0.2.15 c7757280 (unmodified) |
default | 34.0 | 31.2 | 78.1–81.0% | 15.4–17.1% |
0.2.15 c7757280 (unmodified) |
--adapt-swaps 0 |
24.9 | 20.9 | 27.1–41.5% | 37.5–42.7% |
System B (Xeon E5-1650 v3 Haswell, 6 cores for the engine; DDR4; RTX 3060 12 GB at PCIe 3.0 x16; both engines built AVX-only for a like-for-like comparison; 5 pool workers)
| Engine | Setting | Decode tok/s | New-prompt decode | Cache hit | Routed over PCIe |
|---|---|---|---|---|---|
| 0.2.7-based | default | 54.9 | 49.7 | 95.0–96.0% | 0.5–0.8% |
| 0.2.7-based | --adapt-swaps 0 |
45.0 | 37.2 | 72.1–79.4% | 4.0–5.3% |
| 0.2.15 + local commits* | default | 39.7 | 37.8 | 73.6–74.6% | 10.7–10.9% |
| 0.2.15 + local commits* | --adapt-swaps 0 |
30.9 | 26.4 | 25.1–32.1% | 26.7–28.5% |
| 0.2.15 + local commits* | STRATA_ADAPT_FETCH=1 |
44.2 | 40.7 | 79.6–82.9% | 8.0–10.0% |
* 0.2.15 with five local prefill/serve commits ported on top. On System A this build decodes the same as unmodified 0.2.15 (34.6 vs 34.0 tok/s, static-cache hit 30–40% vs 27–42%).
0.2.15 also gets slightly fewer cache slots: expert cache 2513 slots, 4.82 GiB against 2638 slots, 5.06 GiB on 0.2.7. That 5% doesn't explain a static hit rate that falls from ~78% to ~33%.
What I ruled out (System A, 0.2.15)
| Tried | Decode tok/s | Cache hit | Note |
|---|---|---|---|
| default, for reference | 34.6 | 79–81% | |
--pcie-frac 0.35 (auto had picked 0.60) |
34.9 | 74–75% | PCIe share drops to ~7%, decode unchanged |
--lookup-max-t 0 |
34.7 | ~81% | |
--pcie-frac 0.35 --lookup-max-t 0 |
33.8 | 73–74% | |
--adapt-swaps 24 |
34.1 | 69–72% | |
--adapt-every 8 |
33.8 | 73–80% | |
--adapt-async 1 |
35.5 | 79–82% | |
STRATA_ADAPT_FETCH=1 |
41.5 | 86–87% | the only setting that helps (+20%) |
STRATA_ADAPT_FETCH=2 |
40.7 | 85–88% | |
A new profile: tools/collect_profile_traces.py --mode routing on 20 prompts of bench/profile_corpus.json, then tools/make_profile_mix.py |
34.1 | 78% (static 27–34%) | no change from the shipped profile |
Re-pack with 0.2.15's tools/iq_pack.py |
35.2 | 81–82% (static 31–33%) | index.txt and native_experts.txt are byte-identical to the old pack |
STRATA_PREFILL_LEND_PCT=0 |
— | — | not usable: most requests returned no tokens |
Also checked:
- Server: each engine ran with the
serve/server.pyof its own tree. - Profile: both engines read the same
data/expert-profile.bin, with an identical SHA-256. - Our refill patch: the 0.2.7-based build was rerun with its optional async-refill patch off. Default: 55.6 / 96%. Static: 44.8 / 75–81%.
Related
- [Engine] Adaptive tier defaults cost ~10 % code decode on a 2x RTX 3090 split whose cache holds ~45 % of each card's experts #4870: adaptive-tier defaults cost decode on a 2× RTX 3090 split.
- [Engine] 8 GB cards: STRATA_ROUTE_PRIOR=1.5 gives +24% decode and 2.3x expert-cache hits; log the routing choice at startup, consider a per-VRAM default #5000: route prior on 8 GB cards.
Both involve the cache tier, but neither describes the initial resident set getting worse between versions.
Prepared with AI assistance. All numbers are from my own runs, and I can share the full engine logs.
Smallest steps or command that reproduce it
Model: ISTA-DASLab Qwen3.8-Flash-Next-GSQ-RCO-GGUF IQ3_S, both shards, pack iq3_s. Engine arguments (paths shortened):
--pack <packs>/iq3_s --native <models>/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf
--ple-gguf <models>/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf
--expert-profile data/expert-profile.bin --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5
--mtp <mtp>/rt --max-context 131072 --kv int8 --ple-io direct --shared-expert-arena <tmpfs>/strata-iq3s.arena
--kv-resident 20480 --pool-cores 1,2,3,4,5 --mtp-draft-vocab data/draft_vocab_en.bin --conversation-cache-mib 1024
Environment for every run: STRATA_PREFILL_CPU_SHARE=auto STRATA_PREFILL_CPU_SHARE_MAX=4608 STRATA_PF_AHEAD=auto STRATA_KV_PREFETCH=1 STRATA_CKPT_POOL=1 STRATA_CKPT_POOL_READY=4. Some local variables were also set; 0.2.15 doesn't read them.
Steps:
- Start the server:
serve/server.py --engine strata --config <cfg> --port 8090. - Send one 400-token warm-up request.
- Send two greedy chat requests, e.g. "Explain in detail how a modern CPU executes instructions…",
max_tokens400. - Read the
decode expert cache hit ratelines. - Repeat with
--adapt-swaps 0added.
Reproduction status
Reproduced locally
Strata version
Strata 0.2.15 (c7757280), engine built from source with STRATA_ISA_FLOOR=avx (CUDA arch 86); compared against 0.2.7 (5e8a4cc) plus local arena-reuse and prefill commits (the decode expert-cache code is 0.2.7's)
Graphics card, memory and system
RTX 3060 12 GB (driver 595.104.02, CUDA 13 runtime) · System A: Xeon E5-1650 v2, 64 GB DDR3 · System B: Xeon E5-1650 v3, DDR4 · Linux 7.0 (Debian-based), engine in a container. System A's cards use a third-party FP16 rate-unlock kernel module; System B shows the same pattern.
Model, size and settings
ISTA-DASLab Qwen3.8-Flash-Next GSQ-RCO IQ3_S · 131K context · --kv int8 · MTP drafting on · pictures off
Logs
The decode expert cache hit rate lines quoted in the tables come from the engine logs; I can attach the full logs.
Prepared with AI assistance. All numbers are from my own runs, and I can share the full engine logs.
Activity
github-actions commentedon Nov 18, 2026
Thanks for the issue. To keep the tracker readable, every item needs an area prefix in the title and the parts of the issue form filled in. This one is missing:
Edit the issue (title and description) and this note goes away on its own. If it is still missing parts in 5 days it will be closed, which is not a rejection: fix it and comment
/reopen.Details: CONTRIBUTING.md
github-actions commentedon Nov 18, 2026
Thanks, this follows the format now.
notfence commentedon Nov 19, 2026
+1. Reproduced on my RTX 4070.
I won't go into details, but in version 0.2.13.1, the generation rate was about 50–100 tok/s; now, in version 0.2.16, it's 40–75.
At the same time, in version 0.2.13.1, there were about 1,500 experts offloaded to VRAM, while in version 0.2.16, there were ~1,400.
Niko1221 commentedon Nov 20, 2026
Thank you for the detailed tables (decode 55.6 tok/s on a 0.2.7-based build against 34.0 on 0.2.15, cache hit 95.7-96.2% against 78.1-81.0%, routed over PCIe 0.8-1.0% against 15.4-17.1%, RTX 3060 12 GB), and for the second report on an RTX 4070. A resident set that hits 27-42% with adaptation off, against 72-81% before, is something we want to find.
Please run the same test on 0.2.17 (engine 0.1.63) and post the numbers. If it is still slower, add the engine's start log from the first line to READY, including the "expert cache N slots" line of both starts. You could also try
STRATA_NV_CHUNK=0, which keeps the prompt chunk at 8192 so that the prompt lends fewer expert slots. If you can bisect between 0.2.7 and 0.2.15, as you offered, that would help most. The issue stays open.[Docs] план работ 22.11: апстрим-синк, FA-ядро, Qwen3.8
kroll-j commentedon Nov 22, 2026
I think I'm affected by the same thing, I'm suddenly getting around 17 tok/s decode on an RTX 3060 12GB after upgrading to 0.2.18, compared to around 39 tok/s before, on a release from a week or maybe two weeks older.
0xluk commentedon Nov 22, 2026
Thanks for the pointers. I bisected it, and the answer is simpler than a cache bug.
Result: the last fast release is 0.2.7.1, the first slow one is 0.2.8, and the first slow commit is
20a63cf0f("Residency-biased routing (STRATA_ROUTE_PRIOR) is off by default again on 12 GB NVIDIA cards (#4402)"). Its parent is 0.2.7.1 itself. So the drop is the deliberate revert of the 0.2.7 defaultSTRATA_ROUTE_PRIOR=1.5on cards of 13 GiB or less. Our "0.2.7-based" comparison build still had it on (its start log says "residency-biased routing is on by default on this card"). I should have spotted that before filing.Confirmed both ways (System A: Xeon E5-1650 v2, AVX only, 64 GB DDR3, RTX 3060 12 GB at 115 W; ISTA IQ3_S, the issue's arguments; clean upstream AVX builds; 2 × 400-token greedy answers):
46c152a1)20a63cf0fSTRATA_ROUTE_PRIOR=1.5STRATA_ROUTE_PRIOR=0Your other requests:
STRATA_NV_CHUNK=0on 0.2.15: 34.2 / 34.2 tok/s, hit 81-82%, no change.20a63cf0f, about 2506 from 0.2.8 (likelycfa197e21, the reserve for what is allocated after the cache) and 2513 on 0.2.15-0.2.18. It doesn't drive the speed:STRATA_DEFAULTS=026on 0.2.15 gives 2646 slots and is still slow (36.3 / 34.3).20a63cf0f, 0.2.15, 0.2.18; local paths replaced with placeholders). I have the others (0.2.7, 0.2.8, 0.2.17) if useful.Quality: I also ran a small A/B on 0.2.18, PRIOR=0 against 1.5, on two systems: the one above with IQ3_S, and System B (Xeon E5-1650 v3, 2× RTX 3060 12 GB at 130 W) with UD-Q4_K_XL v3.1. It used 12 long generations up to 2000 tokens, 12 arithmetic items and 8 small functions run against tests, thinking off, one sample per item. With 1.5 I saw no clear loss: no loops (one with PRIOR=0), similar distinct-4-gram ratios, all functions that had a body passed, and 2 arithmetic misses out of 24 against 0 out of 24 (not significant at this size). Generation was 52-98% faster. This does not cover long agent sessions, which is where #4402 saw the loops, so I can't confirm or rule out that finding.
From my side the issue can be closed as "intended default", unless you want to track the 0.2.15 → 0.2.17 slowdown here. A note in the docs that 12 GB cards lose ~35% decode without
STRATA_ROUTE_PRIOR=1.5would help others who upgrade from 0.2.7. Thanks.Start log 0.2.7.1 (fast)
Start log 20a63cf (first slow)
Start log 0.2.15
Start log 0.2.18
Prepared with AI assistance; all numbers are from my own runs.