Skip to content

[Engine] 0.2.15 decodes 28-36% slower than 0.2.7 on RTX 3060 12 GB: --expert-cache auto starts with a much worse resident set (same pack and profile) #6208

Open

Description

@0xluk

Area

[Engine] (speed, kernels, models, the server and API)

Desktop app, Terminal or source?

Source (START-HERE.bat, setup.sh or a build)

What happened

With the same model files, the same pack and the same expert profile, 0.2.15 decodes 28–36% slower than a 0.2.7-based build on an RTX 3060 12 GB. The decode expert-cache hit rate falls from ~96% to ~75–81%. The clearest signal is with adaptation off (--adapt-swaps 0): the resident set that --expert-cache auto builds from the profile hits 25–42% of routed experts on 0.2.15, against 72–81% on 0.2.7. Swapping then has to repair a much worse starting set, and it only gets part of the way.

I have not bisected it yet, so I don't know which commit changed this. I can bisect if that helps.

Measurements

1 agent, IQ3_S, one GPU, 131K context, --kv int8, MTP drafting on. "decode" = two fixed 400-token greedy chat answers. "new-prompt decode" = two 300-token answers on topics that are new in every run. Speed = (completion tokens − 1) / (last token time − first token time). Hit rate and PCIe share come from the server's decode expert cache hit rate lines. 1 run per row.

System A (Xeon E5-1650 v2 Ivy Bridge, AVX only; 64 GB DDR3-1600; RTX 3060 12 GB at PCIe 3.0 x16, capped at 115 W; 5 pool workers)

Engine Setting Decode tok/s New-prompt decode Cache hit Routed over PCIe
0.2.7-based default 55.6 52.6 95.7–96.2% 0.8–1.0%
0.2.7-based --adapt-swaps 0 44.8 36.8 75.0–81.3% 5.0–6.6%
0.2.15 c7757280 (unmodified) default 34.0 31.2 78.1–81.0% 15.4–17.1%
0.2.15 c7757280 (unmodified) --adapt-swaps 0 24.9 20.9 27.1–41.5% 37.5–42.7%

System B (Xeon E5-1650 v3 Haswell, 6 cores for the engine; DDR4; RTX 3060 12 GB at PCIe 3.0 x16; both engines built AVX-only for a like-for-like comparison; 5 pool workers)

Engine Setting Decode tok/s New-prompt decode Cache hit Routed over PCIe
0.2.7-based default 54.9 49.7 95.0–96.0% 0.5–0.8%
0.2.7-based --adapt-swaps 0 45.0 37.2 72.1–79.4% 4.0–5.3%
0.2.15 + local commits* default 39.7 37.8 73.6–74.6% 10.7–10.9%
0.2.15 + local commits* --adapt-swaps 0 30.9 26.4 25.1–32.1% 26.7–28.5%
0.2.15 + local commits* STRATA_ADAPT_FETCH=1 44.2 40.7 79.6–82.9% 8.0–10.0%

* 0.2.15 with five local prefill/serve commits ported on top. On System A this build decodes the same as unmodified 0.2.15 (34.6 vs 34.0 tok/s, static-cache hit 30–40% vs 27–42%).

0.2.15 also gets slightly fewer cache slots: expert cache 2513 slots, 4.82 GiB against 2638 slots, 5.06 GiB on 0.2.7. That 5% doesn't explain a static hit rate that falls from ~78% to ~33%.

What I ruled out (System A, 0.2.15)

Tried Decode tok/s Cache hit Note
default, for reference 34.6 79–81%
--pcie-frac 0.35 (auto had picked 0.60) 34.9 74–75% PCIe share drops to ~7%, decode unchanged
--lookup-max-t 0 34.7 ~81%
--pcie-frac 0.35 --lookup-max-t 0 33.8 73–74%
--adapt-swaps 24 34.1 69–72%
--adapt-every 8 33.8 73–80%
--adapt-async 1 35.5 79–82%
STRATA_ADAPT_FETCH=1 41.5 86–87% the only setting that helps (+20%)
STRATA_ADAPT_FETCH=2 40.7 85–88%
A new profile: tools/collect_profile_traces.py --mode routing on 20 prompts of bench/profile_corpus.json, then tools/make_profile_mix.py 34.1 78% (static 27–34%) no change from the shipped profile
Re-pack with 0.2.15's tools/iq_pack.py 35.2 81–82% (static 31–33%) index.txt and native_experts.txt are byte-identical to the old pack
STRATA_PREFILL_LEND_PCT=0 — — not usable: most requests returned no tokens

Also checked:

  • Server: each engine ran with the serve/server.py of its own tree.
  • Profile: both engines read the same data/expert-profile.bin, with an identical SHA-256.
  • Our refill patch: the 0.2.7-based build was rerun with its optional async-refill patch off. Default: 55.6 / 96%. Static: 44.8 / 75–81%.

Related

Both involve the cache tier, but neither describes the initial resident set getting worse between versions.

Prepared with AI assistance. All numbers are from my own runs, and I can share the full engine logs.

Smallest steps or command that reproduce it

Model: ISTA-DASLab Qwen3.8-Flash-Next-GSQ-RCO-GGUF IQ3_S, both shards, pack iq3_s. Engine arguments (paths shortened):

--pack <packs>/iq3_s --native <models>/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf
--ple-gguf <models>/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf
--expert-profile data/expert-profile.bin --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5
--mtp <mtp>/rt --max-context 131072 --kv int8 --ple-io direct --shared-expert-arena <tmpfs>/strata-iq3s.arena
--kv-resident 20480 --pool-cores 1,2,3,4,5 --mtp-draft-vocab data/draft_vocab_en.bin --conversation-cache-mib 1024

Environment for every run: STRATA_PREFILL_CPU_SHARE=auto STRATA_PREFILL_CPU_SHARE_MAX=4608 STRATA_PF_AHEAD=auto STRATA_KV_PREFETCH=1 STRATA_CKPT_POOL=1 STRATA_CKPT_POOL_READY=4. Some local variables were also set; 0.2.15 doesn't read them.

Steps:

  1. Start the server: serve/server.py --engine strata --config <cfg> --port 8090.
  2. Send one 400-token warm-up request.
  3. Send two greedy chat requests, e.g. "Explain in detail how a modern CPU executes instructions…", max_tokens 400.
  4. Read the decode expert cache hit rate lines.
  5. Repeat with --adapt-swaps 0 added.

Reproduction status

Reproduced locally

Strata version

Strata 0.2.15 (c7757280), engine built from source with STRATA_ISA_FLOOR=avx (CUDA arch 86); compared against 0.2.7 (5e8a4cc) plus local arena-reuse and prefill commits (the decode expert-cache code is 0.2.7's)

Graphics card, memory and system

RTX 3060 12 GB (driver 595.104.02, CUDA 13 runtime) · System A: Xeon E5-1650 v2, 64 GB DDR3 · System B: Xeon E5-1650 v3, DDR4 · Linux 7.0 (Debian-based), engine in a container. System A's cards use a third-party FP16 rate-unlock kernel module; System B shows the same pattern.

Model, size and settings

ISTA-DASLab Qwen3.8-Flash-Next GSQ-RCO IQ3_S · 131K context · --kv int8 · MTP drafting on · pictures off

Logs

The decode expert cache hit rate lines quoted in the tables come from the engine logs; I can attach the full logs.

Prepared with AI assistance. All numbers are from my own runs, and I can share the full engine logs.

Activity

  1. github-actions commented on Nov 18, 2026

    @github-actions

    Thanks for the issue. To keep the tracker readable, every item needs an area prefix in the title and the parts of the issue form filled in. This one is missing:

    • Fill in "Area".
    • Fill in "Desktop app, Terminal or source?".
    • Fill in "Reproduction status".
    • Fill in "Strata version".
    • Fill in "Graphics card, memory and system".

    Edit the issue (title and description) and this note goes away on its own. If it is still missing parts in 5 days it will be closed, which is not a rejection: fix it and comment /reopen.

    Details: CONTRIBUTING.md

  2. github-actions commented on Nov 18, 2026

    @github-actions

    Thanks, this follows the format now.

  3. notfence commented on Nov 19, 2026

    @notfence

    +1. Reproduced on my RTX 4070.
    I won't go into details, but in version 0.2.13.1, the generation rate was about 50–100 tok/s; now, in version 0.2.16, it's 40–75.
    At the same time, in version 0.2.13.1, there were about 1,500 experts offloaded to VRAM, while in version 0.2.16, there were ~1,400.

  4. Niko1221 commented on Nov 20, 2026

    @Niko1221
    Owner

    Thank you for the detailed tables (decode 55.6 tok/s on a 0.2.7-based build against 34.0 on 0.2.15, cache hit 95.7-96.2% against 78.1-81.0%, routed over PCIe 0.8-1.0% against 15.4-17.1%, RTX 3060 12 GB), and for the second report on an RTX 4070. A resident set that hits 27-42% with adaptation off, against 72-81% before, is something we want to find.

    Please run the same test on 0.2.17 (engine 0.1.63) and post the numbers. If it is still slower, add the engine's start log from the first line to READY, including the "expert cache N slots" line of both starts. You could also try STRATA_NV_CHUNK=0, which keeps the prompt chunk at 8192 so that the prompt lends fewer expert slots. If you can bisect between 0.2.7 and 0.2.15, as you offered, that would help most. The issue stays open.

  5. kroll-j commented on Nov 22, 2026

    @kroll-j

    I think I'm affected by the same thing, I'm suddenly getting around 17 tok/s decode on an RTX 3060 12GB after upgrading to 0.2.18, compared to around 39 tok/s before, on a release from a week or maybe two weeks older.

  6. 0xluk commented on Nov 22, 2026

    @0xluk
    Author

    Thanks for the pointers. I bisected it, and the answer is simpler than a cache bug.

    Result: the last fast release is 0.2.7.1, the first slow one is 0.2.8, and the first slow commit is 20a63cf0f ("Residency-biased routing (STRATA_ROUTE_PRIOR) is off by default again on 12 GB NVIDIA cards (#4402)"). Its parent is 0.2.7.1 itself. So the drop is the deliberate revert of the 0.2.7 default STRATA_ROUTE_PRIOR=1.5 on cards of 13 GiB or less. Our "0.2.7-based" comparison build still had it on (its start log says "residency-biased routing is on by default on this card"). I should have spotted that before filing.

    Confirmed both ways (System A: Xeon E5-1650 v2, AVX only, 64 GB DDR3, RTX 3060 12 GB at 115 W; ISTA IQ3_S, the issue's arguments; clean upstream AVX builds; 2 × 400-token greedy answers):

    Build Decode tok/s Decode cache hit
    0.2.7.1 (46c152a1) 54.7 / 62.6 94.9-96.2%
    20a63cf0f 38.2 / 34.6 75.5-82.0%
    0.2.8 35.8 / 32.9 75.9-76.1%
    0.2.15 33.8 / 34.0 79.2-82.2%
    0.2.15 + STRATA_ROUTE_PRIOR=1.5 54.3 / 54.5 96.9-97.0%
    0.2.7.1 + STRATA_ROUTE_PRIOR=0 37.4 / 34.0 76.1-80.4%

    Your other requests:

    • 0.2.17 (engine 0.1.63): 30.2 / 28.3 tok/s, hit 72.5-77.9%. 0.2.18 (0.1.64): 28.3 / 28.0, hit 73.5-74.6%, PCIe share 15-17%. That is about 15% below 0.2.15 (~34) on the same setup, so there seems to be a second, smaller slowdown after 0.2.15. It may be what @kroll-j sees. I have not bisected it.
    • STRATA_NV_CHUNK=0 on 0.2.15: 34.2 / 34.2 tok/s, hit 81-82%, no change.
    • Slots: 2638 up to 20a63cf0f, about 2506 from 0.2.8 (likely cfa197e21, the reserve for what is allocated after the cache) and 2513 on 0.2.15-0.2.18. It doesn't drive the speed: STRATA_DEFAULTS=026 on 0.2.15 gives 2646 slots and is still slow (36.3 / 34.3).
    • Start logs from the first line to READY are below (0.2.7.1, 20a63cf0f, 0.2.15, 0.2.18; local paths replaced with placeholders). I have the others (0.2.7, 0.2.8, 0.2.17) if useful.

    Quality: I also ran a small A/B on 0.2.18, PRIOR=0 against 1.5, on two systems: the one above with IQ3_S, and System B (Xeon E5-1650 v3, 2× RTX 3060 12 GB at 130 W) with UD-Q4_K_XL v3.1. It used 12 long generations up to 2000 tokens, 12 arithmetic items and 8 small functions run against tests, thinking off, one sample per item. With 1.5 I saw no clear loss: no loops (one with PRIOR=0), similar distinct-4-gram ratios, all functions that had a body passed, and 2 arithmetic misses out of 24 against 0 out of 24 (not significant at this size). Generation was 52-98% faster. This does not cover long agent sessions, which is where #4402 saw the loops, so I can't confirm or rule out that finding.

    From my side the issue can be closed as "intended default", unless you want to track the 0.2.15 → 0.2.17 slowdown here. A note in the docs that 12 GB cards lose ~35% decode without STRATA_ROUTE_PRIOR=1.5 would help others who upgrade from 0.2.7. Thanks.

    Start log 0.2.7.1 (fast)
    [strata] 2026-11-22 12:49:59 engine started: ./strata --serve --pack <data>/packs/iq3_s --native <model-dir>/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf --ple-gguf <model-dir>/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf --expert-profile <data>/expert-profile.bin --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp <data>/mtp/rt --max-context 131072 --kv int8 --ple-io direct --shared-expert-arena <tmpfs>/arena --kv-resident 20480 --pool-cores 1,2,3,4,5 --mtp-draft-vocab data/draft_vocab_en.bin --conversation-cache-mib 1024
    strata: sm_86 (Ampere): 9 exact speed switches on by default (STRATA_SM86_PERF=0 turns them off; a switch you set is kept): ADAPT_NOWAIT ADAPT_LAG CPU_IQ_ALL_EXACT1 SIDE_MIXER ATTN_DAILY GR_UP75 GU_SWIGLU QSA_ROWS B6_MMVQ_ROWS
    strata generate: memory guard: limit 58.0 GiB (source: meminfo), current 0.5 GiB, reclaimable cache credited 0.0 GiB; available 57.5 GiB (MemAvailable 57.5 GiB), floor 2560 MiB
    strata generate: EXPERIMENTAL older-CPU build (ggml-cpu for avx): this CPU has no AVX2, so the CPU's experts run on ggml-cpu's kernels - expect it to be slow
    strata generate: PCIe probe: 11.2 GB/s host->device (best of 11.2 11.1 10.9 10.9) -> pcie_frac 0.31 (default 0.55)
    strata generate: this CPU has no AVX-512: the expert kernels run on AVX1 128-bit (iq_avx1, the older-CPU build) (multi-token for the i-quant gate/up rows)
    strata generate: native pack: <data>/packs/iq3_s experts (largest blob 2.66 MB), token embedding IQ4_XS in mapped host memory (322 MiB, 0.6 s)
    strata generate: 1466 MiB of weights loaded from <data>/packs/iq3_s in 0.6 s (302 canonical tensors skipped: served natively)
    strata generate: 300 native projection matrices, 2018.88 MiB of weights, in 3.9 s
    strata generate: PLE direct read probe: 75694 rows/s (fine)
    strata generate: the SSD is kept awake while rows are read: one page of the table after 100 ms without a read, until 60 s after the last request (STRATA_SSD_KEEPALIVE=0 turns it off)
    strata generate: profile <data>/expert-profile.bin: 24576 ranked pairs, built for 24576 slots
    strata generate: KV streaming: 20480 of 131072 cells per QSA layer in VRAM, the K/V in 1.55 GiB of pinned RAM
    strata generate: PLE on, table 320001536 rows of <model-dir>/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf
    strata mtp: draft layer loaded, 835 MiB of VRAM (experts 675, dense 111), files read in 0.53 s (1489 MiB/s)
    strata generate: experimental native Q5_K head, 521472000 bytes, in 1.0 s
    strata generate: GPU 0: NVIDIA GeForce RTX 3060, compute capability 8.6
    strata generate: expert arena read through the file cache (buffered (not Windows))
    strata generate: expert arena: cudaHostRegister PORTABLE ok; MAP_SHARED file <tmpfs>/arena (pack hash checked)
    strata generate: loaded 46.84 GiB at 0.53 GiB/s
    strata generate: GPU host-callback thread started before host-core pinning
    strata generate: CPU pool tasks/phase: 18 (automatic), participating threads: 6
    strata generate: 5 pool workers on logical processors 1,2,3,4,5, the host thread on 0 (draining too) (--host-core first)
    strata generate: CPU workers on logical processors 1-5 (5 workers, your --pool-cores list, one per physical core; host on 0)
    strata aux cpus: threads that are neither pool workers nor the host go to CPUs 6 (off at start; a request's aux_cpus=1 turns it on)
    strata generate: expert cache auto: 5.83 GiB free, 700 MiB reserved (+86 MiB for the draft head) -> 2041 slots
    strata generate: expert cache 2638 slots, 5.06 GiB of VRAM; policy is
    strata generate: the GPU computes the experts in the cache; it rounds differently from the CPU,
                     so a reply can differ slightly from a run without the cache (same quality:
                     bench/results/2026-09-27-cache-parity).
                     PROFILE, ranked by routing frequency, no eviction.
    strata generate: pre-filled 2638 of 2638 slots from the profile in 0.5 s (10755 MB/s); slot 0 verified
    strata generate: R4 hit path ON - resident experts are computed on the GPU
    strata generate: 5 expert-pool workers + the host thread
    strata generate: session is up (engine 0.1.53)
    strata generate: token graph hit path: 2638 resident experts, decided on the device
    strata serve: prompt chunk auto: 8192 tokens, a 384-slot ring
    strata serve: the prompt path borrows 2219 CUDA0 cache slots (4.25 GiB)
    strata verify: residency-biased routing is on by default on this card (STRATA_ROUTE_PRIOR=1.5: a card of 13 GiB or less; it changes which experts run; STRATA_ROUTE_PRIOR=0 turns it off)
    strata hc: CUDA0: the hyper-connection read runs as staged (the norm per token and stream, the down projection's activations staged ahead by cp.async); checked bit for bit against the plain read on this card (STRATA_HC_SPLIT=1 or 0 for the earlier ones)
    strata verify: window up to 6 tokens, 75.1 MiB of device buffers
    strata mtp: draft head over 40525 tokens (81.2 MiB)
    strata serve: STRATA_ROUTE_TAIL_SKIP=7 is ON (default): a missed expert that every token of a verify window routes at rank 7 or lower is skipped (+10..20% decode on 12-16 GB cards that miss experts, answers differ slightly from 0.1.41); STRATA_ROUTE_TAIL_SKIP=0 turns it off
    strata serve: adaptive overlap: off: not with STRATA_ADAPT_LAG / STRATA_ADAPT_NOWAIT
    strata serve: the adaptive tier's evictions reach the device's table before the next window (STRATA_ADAPT_EVICT_SYNC=0: when its copies land)
    strata serve: adaptive tier: blocking, and the asynchronous tier is tried once the session shows the swaps hold the windows up (--adapt-async 0 or STRATA_ADAPT_ASYNC=0: never)
    strata serve: 456 MiB of VRAM free with everything loaded
    
    Start log 20a63cf (first slow)
    [strata] 2026-11-22 13:04:14 engine started: ./strata --serve --pack <data>/packs/iq3_s --native <model-dir>/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf --ple-gguf <model-dir>/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf --expert-profile <data>/expert-profile.bin --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp <data>/mtp/rt --max-context 131072 --kv int8 --ple-io direct --shared-expert-arena <tmpfs>/arena --kv-resident 20480 --pool-cores 1,2,3,4,5 --mtp-draft-vocab data/draft_vocab_en.bin --conversation-cache-mib 1024
    strata: sm_86 (Ampere): 9 exact speed switches on by default (STRATA_SM86_PERF=0 turns them off; a switch you set is kept): ADAPT_NOWAIT ADAPT_LAG CPU_IQ_ALL_EXACT1 SIDE_MIXER ATTN_DAILY GR_UP75 GU_SWIGLU QSA_ROWS B6_MMVQ_ROWS
    strata generate: memory guard: limit 58.0 GiB (source: meminfo), current 0.5 GiB, reclaimable cache credited 0.0 GiB; available 57.5 GiB (MemAvailable 57.5 GiB), floor 2560 MiB
    strata generate: EXPERIMENTAL older-CPU build (ggml-cpu for avx): this CPU has no AVX2, so the CPU's experts run on ggml-cpu's kernels - expect it to be slow
    strata generate: PCIe probe: 11.2 GB/s host->device (best of 11.1 11.1 11.2 11.2) -> pcie_frac 0.31 (default 0.55)
    strata generate: this CPU has no AVX-512: the expert kernels run on AVX1 128-bit (iq_avx1, the older-CPU build) (multi-token for the i-quant gate/up rows)
    strata generate: native pack: <data>/packs/iq3_s experts (largest blob 2.66 MB), token embedding IQ4_XS in mapped host memory (322 MiB, 0.6 s)
    strata generate: 1466 MiB of weights loaded from <data>/packs/iq3_s in 0.6 s (302 canonical tensors skipped: served natively)
    strata generate: 300 native projection matrices, 2018.88 MiB of weights, in 4.0 s
    strata generate: PLE direct read probe: 76089 rows/s (fine)
    strata generate: the SSD is kept awake while rows are read: one page of the table after 100 ms without a read, until 60 s after the last request (STRATA_SSD_KEEPALIVE=0 turns it off)
    strata generate: profile <data>/expert-profile.bin: 24576 ranked pairs, built for 24576 slots
    strata generate: KV streaming: 20480 of 131072 cells per QSA layer in VRAM, the K/V in 1.55 GiB of pinned RAM
    strata generate: PLE on, table 320001536 rows of <model-dir>/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf
    strata mtp: draft layer loaded, 835 MiB of VRAM (experts 675, dense 111), files read in 0.52 s (1508 MiB/s)
    strata generate: experimental native Q5_K head, 521472000 bytes, in 1.0 s
    strata generate: GPU 0: NVIDIA GeForce RTX 3060, compute capability 8.6
    strata generate: expert arena read through the file cache (buffered (not Windows))
    strata generate: expert arena: cudaHostRegister PORTABLE ok; MAP_SHARED file <tmpfs>/arena (pack hash checked)
    strata generate: loaded 46.84 GiB at 0.53 GiB/s
    strata generate: GPU host-callback thread started before host-core pinning
    strata generate: CPU pool tasks/phase: 18 (automatic), participating threads: 6
    strata generate: 5 pool workers on logical processors 1,2,3,4,5, the host thread on 0 (draining too) (--host-core first)
    strata generate: CPU workers on logical processors 1-5 (5 workers, your --pool-cores list, one per physical core; host on 0)
    strata aux cpus: threads that are neither pool workers nor the host go to CPUs 6 (off at start; a request's aux_cpus=1 turns it on)
    strata generate: expert cache auto: 5.83 GiB free, 700 MiB reserved (+86 MiB for the draft head) -> 2041 slots
    strata generate: expert cache 2638 slots, 5.06 GiB of VRAM; policy is
    strata generate: the GPU computes the experts in the cache; it rounds differently from the CPU,
                     so a reply can differ slightly from a run without the cache (same quality:
                     bench/results/2026-09-27-cache-parity).
                     PROFILE, ranked by routing frequency, no eviction.
    strata generate: pre-filled 2638 of 2638 slots from the profile in 0.5 s (10637 MB/s); slot 0 verified
    strata generate: R4 hit path ON - resident experts are computed on the GPU
    strata generate: 5 expert-pool workers + the host thread
    strata generate: session is up (engine 0.1.53)
    strata generate: token graph hit path: 2638 resident experts, decided on the device
    strata serve: prompt chunk auto: 8192 tokens, a 384-slot ring
    strata serve: the prompt path borrows 2219 CUDA0 cache slots (4.25 GiB)
    strata hc: CUDA0: the hyper-connection read runs as staged (the norm per token and stream, the down projection's activations staged ahead by cp.async); checked bit for bit against the plain read on this card (STRATA_HC_SPLIT=1 or 0 for the earlier ones)
    strata verify: window up to 6 tokens, 75.1 MiB of device buffers
    strata mtp: draft head over 40525 tokens (81.2 MiB)
    strata serve: STRATA_ROUTE_TAIL_SKIP=7 is ON (default): a missed expert that every token of a verify window routes at rank 7 or lower is skipped (+10..20% decode on 12-16 GB cards that miss experts, answers differ slightly from 0.1.41); STRATA_ROUTE_TAIL_SKIP=0 turns it off
    strata serve: adaptive overlap: off: not with STRATA_ADAPT_LAG / STRATA_ADAPT_NOWAIT
    strata serve: the adaptive tier's evictions reach the device's table before the next window (STRATA_ADAPT_EVICT_SYNC=0: when its copies land)
    strata serve: adaptive tier: blocking, and the asynchronous tier is tried once the session shows the swaps hold the windows up (--adapt-async 0 or STRATA_ADAPT_ASYNC=0: never)
    strata serve: 456 MiB of VRAM free with everything loaded
    
    Start log 0.2.15
    [strata] 2026-11-22 10:10:37 engine started: ./strata --serve --pack <data>/packs/iq3_s --native <model-dir>/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf --ple-gguf <model-dir>/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf --expert-profile <data>/expert-profile.bin --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp <data>/mtp/rt --max-context 131072 --kv int8 --ple-io direct --shared-expert-arena <tmpfs>/arena --kv-resident 20480 --pool-cores 1,2,3,4,5 --mtp-draft-vocab data/draft_vocab_en.bin --conversation-cache-mib 1024
    strata: sm_86 (Ampere): 8 exact speed switches on by default (STRATA_SM86_PERF=0 turns them off; a switch you set is kept): ADAPT_LAG CPU_IQ_ALL_EXACT1 SIDE_MIXER ATTN_DAILY GR_UP75 GU_SWIGLU QSA_ROWS B6_MMVQ_ROWS
    strata generate: NVIDIA: --prefill auto may take chunks up to 32768 (STRATA_NV_CHUNK=0 keeps 8192, =16 keeps 16384)
    strata generate: memory guard: limit 58.0 GiB (source: meminfo), current 0.5 GiB, reclaimable cache credited 0.0 GiB; available 57.5 GiB (MemAvailable 57.5 GiB), floor 2560 MiB
    strata generate: EXPERIMENTAL older-CPU build (ggml-cpu for avx): this CPU has no AVX2, so the CPU's experts run on ggml-cpu's kernels - expect it to be slow
    strata generate: PCIe probe: 11.2 GB/s host->device (best of 11.2 11.1 11.2 11.2) -> pcie_frac 0.31 (default 0.55)
    strata generate: CPU expert kernels: AVX1 (older-CPU build)
    strata generate: native pack: <data>/packs/iq3_s experts (largest blob 2.66 MB), token embedding IQ4_XS in mapped host memory (322 MiB, 0.6 s)
    strata generate: 1466 MiB of weights loaded from <data>/packs/iq3_s in 3.8 s (302 canonical tensors skipped: served natively)
    strata generate: 300 native projection matrices, 2018.88 MiB of weights, in 3.9 s
    strata generate: PLE direct read probe: 77757 rows/s (fine)
    strata generate: the SSD is kept awake while rows are read: one page of the table after 100 ms without a read, until 60 s after the last request (STRATA_SSD_KEEPALIVE=0 turns it off)
    strata generate: profile <data>/expert-profile.bin: 24576 ranked pairs, built for 24576 slots
    strata generate: KV streaming: 20480 of 131072 cells per QSA layer in VRAM, the K/V in 1.55 GiB of pinned RAM
    strata generate: PLE on, table 320001536 rows of <model-dir>/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf
    strata mtp: draft layer loaded, 837 MiB of VRAM (experts 675, dense 111), files read in 0.53 s (1484 MiB/s)
    strata generate: experimental native Q5_K head, 521472000 bytes, in 1.0 s
    strata generate: GPU 0: NVIDIA GeForce RTX 3060, compute capability 8.6
    strata generate: expert arena read through the file cache (buffered (not Windows))
    strata generate: expert arena: cudaHostRegister PORTABLE ok; MAP_SHARED file <tmpfs>/arena (pack hash checked)
    strata generate: loaded 46.84 GiB at 0.52 GiB/s
    strata generate: GPU host-callback thread started before host-core pinning
    strata generate: CPU pool tasks/phase: 18 (automatic), participating threads: 6
    strata generate: 5 pool workers on logical processors 1,2,3,4,5, the host thread on 0 (draining too) (--host-core first)
    strata generate: CPU workers on logical processors 1-5 (5 workers, your --pool-cores list, one per physical core; host on 0)
    strata aux cpus: threads that are neither pool workers nor the host go to CPUs 6 (off at start; a request's aux_cpus=1 turns it on)
    strata generate: expert cache auto: 5.85 GiB free, 700 MiB reserved (+88 MiB for the draft head, +256 MiB for what is allocated after the cache) -> 1946 slots
    strata generate: expert cache 2513 slots, 4.82 GiB of VRAM; policy is
    strata generate: the GPU computes the experts in the cache; it rounds differently from the CPU,
                     so a reply can differ slightly from a run without the cache (same quality:
                     bench/results/2026-09-27-cache-parity).
                     PROFILE, ranked by routing frequency, no eviction.
    strata generate: pre-filled 2513 of 2513 slots from the profile in 0.5 s (10696 MB/s); slot 0 verified
    strata generate: R4 hit path ON - resident experts are computed on the GPU
    strata generate: 5 expert-pool workers + the host thread
    strata generate: session is up (engine 0.1.61)
    strata generate: token graph hit path: 2513 resident experts, decided on the device
    strata: prompt kernels: chunks of 4608 tokens or more run 48 of 48 layers on the fused kernels; shorter chunks run 48 on the staged fused entry.
    strata serve: small expert cache: the default prompt path would read in chunks of 7680 tokens (under 8192), so prompts are read the way that suits a card with little memory and, from two chunks, the layer-major walk (chunk 7168 tokens); STRATA_PF_SMALL=0 keeps the default path, STRATA_PF_FUSED / STRATA_PREFILL_LAYER_MAJOR set by hand win
    strata serve: prompt chunk auto: 7168 tokens, a 384-slot ring
    strata serve: the prompt path borrows 2223 CUDA0 cache slots (4.26 GiB)
    strata hc: CUDA0: the hyper-connection read runs as staged (the norm per token and stream, the down projection's activations staged ahead by cp.async); checked bit for bit against the plain read on this card (STRATA_HC_SPLIT=1 or 0 for the earlier ones)
    strata verify: window up to 12 tokens, 109.5 MiB of device buffers
    strata mtp: draft head over 40525 tokens (81.2 MiB)
    strata serve: PCIe share: start 0.30 from 5 pool workers (assumed 20 GB/s of expert bytes) and the 11.2 GB/s link (was 0.31)
    strata serve: STRATA_ROUTE_TAIL_SKIP=7 is ON (default): a missed expert that every token of a verify window routes at rank 7 or lower is skipped (+10..20% decode on 12-16 GB cards that miss experts, answers differ slightly from 0.1.41); STRATA_ROUTE_TAIL_SKIP=0 turns it off
    strata serve: adaptive overlap: off: not with STRATA_ADAPT_LAG / STRATA_ADAPT_NOWAIT
    strata serve: the adaptive tier's evictions reach the device's table before the next window (STRATA_ADAPT_EVICT_SYNC=0: when its copies land)
    strata serve: adaptive tier: blocking, and the asynchronous tier is tried once the session shows the swaps hold the windows up (--adapt-async 0 or STRATA_ADAPT_ASYNC=0: never)
    strata serve: 680 MiB of VRAM free with everything loaded
    strata verify: capturing the 1-token window (680 MiB of VRAM free)
    strata verify: captured the 1-token window (upload no error, sync no error) in 41.2 ms
    strata verify: capturing the 2-token window (666 MiB of VRAM free)
    strata verify: captured the 2-token window (upload no error, sync no error) in 19.1 ms
    strata verify: capturing the 3-token window (652 MiB of VRAM free)
    strata verify: captured the 3-token window (upload no error, sync no error) in 18.9 ms
    strata verify: capturing the 4-token window (638 MiB of VRAM free)
    strata verify: captured the 4-token window (upload no error, sync no error) in 19.5 ms
    strata verify: capturing the 5-token window (622 MiB of VRAM free)
    strata verify: captured the 5-token window (upload no error, sync no error) in 17.0 ms
    strata verify: capturing the 6-token window (608 MiB of VRAM free)
    strata verify: captured the 6-token window (upload no error, sync no error) in 17.2 ms
    strata verify: capturing the 12-token window (594 MiB of VRAM free)
    strata verify: captured the 12-token window (upload no error, sync no error) in 40.5 ms
    strata serve: window warm-up: ms per width 1:68 2:97 3:115 4:149 5:181 6:219 12:388, total 1392 ms
    strata serve: --lookup-max-t 12: MTP windows up to 4 tokens, window sizes 1,2,3,4,5,6,12
    
    Start log 0.2.18
    [strata] 2026-11-22 13:45:23 engine started: ./strata --serve --pack <data>/packs/iq3_s --native <model-dir>/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf --ple-gguf <model-dir>/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf --expert-profile <data>/expert-profile.bin --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp <data>/mtp/rt --max-context 131072 --kv int8 --ple-io direct --shared-expert-arena <tmpfs>/arena --kv-resident 20480 --pool-cores 1,2,3,4,5 --mtp-draft-vocab data/draft_vocab_en.bin --conversation-cache-mib 1024
    strata: sm_86 (Ampere): 8 exact speed switches on by default (STRATA_SM86_PERF=0 turns them off; a switch you set is kept): ADAPT_LAG CPU_IQ_ALL_EXACT1 SIDE_MIXER ATTN_DAILY GR_UP75 GU_SWIGLU QSA_ROWS B6_MMVQ_ROWS
    strata generate: NVIDIA: --prefill auto may take chunks up to 32768 (STRATA_NV_CHUNK=0 keeps 8192, =16 keeps 16384)
    strata generate: memory guard: limit 58.0 GiB (source: meminfo), current 0.3 GiB, reclaimable cache credited 0.0 GiB; available 57.7 GiB (MemAvailable 57.7 GiB), floor 2560 MiB
    strata generate: EXPERIMENTAL older-CPU build (ggml-cpu for avx): this CPU has no AVX2, so the CPU's experts run on ggml-cpu's kernels - expect it to be slow
    strata generate: PCIe probe: 11.1 GB/s host->device (best of 11.1 11.1 10.9 11.0) -> pcie_frac 0.31 (default 0.55)
    strata generate: CPU expert kernels: AVX1 (older-CPU build)
    strata generate: native pack: <data>/packs/iq3_s experts (largest blob 2.66 MB), token embedding IQ4_XS in mapped host memory (322 MiB, 0.6 s)
    strata generate: 1466 MiB of weights loaded from <data>/packs/iq3_s in 1.9 s (302 canonical tensors skipped: served natively)
    strata generate: 300 native projection matrices, 2018.88 MiB of weights, in 3.9 s
    strata generate: PLE direct read probe: 79107 rows/s (fine)
    strata generate: the SSD is kept awake while rows are read: one page of the table after 100 ms without a read, until 60 s after the last request (STRATA_SSD_KEEPALIVE=0 turns it off)
    strata generate: profile <data>/expert-profile.bin: 24576 ranked pairs, built for 24576 slots
    strata generate: KV streaming: 20480 of 131072 cells per QSA layer in VRAM, the K/V in 1.55 GiB of pinned RAM
    strata generate: PLE on, table 320001536 rows of <model-dir>/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf
    strata mtp: draft layer loaded, 837 MiB of VRAM (experts 675, dense 111), files read in 0.52 s (1525 MiB/s)
    strata generate: experimental native Q6_K head, 521472000 bytes, in 1.0 s
    strata generate: GPU 0: NVIDIA GeForce RTX 3060, compute capability 8.6
    strata generate: expert arena read through the file cache (buffered (not Windows))
    strata generate: expert arena: cudaHostRegister PORTABLE ok; MAP_SHARED file <tmpfs>/arena (pack hash checked)
    strata generate: loaded 46.84 GiB at 0.52 GiB/s
    strata generate: GPU host-callback thread started before host-core pinning
    strata generate: CPU pool tasks/phase: 18 (automatic), participating threads: 6
    strata generate: 5 pool workers on logical processors 1,2,3,4,5, the host thread on 0 (draining too) (--host-core first)
    strata generate: CPU workers on logical processors 1-5 (5 workers, your --pool-cores list, one per physical core; host on 0)
    strata aux cpus: threads that are neither pool workers nor the host go to CPUs 6 (off at start; a request's aux_cpus=1 turns it on)
    strata generate: expert cache auto: 5.85 GiB free, 700 MiB reserved (+88 MiB for the draft head, +256 MiB for what is allocated after the cache) -> 1946 slots
    strata generate: expert cache 2513 slots, 4.82 GiB of VRAM; policy is
    strata generate: the GPU computes the experts in the cache; it rounds differently from the CPU,
                     so a reply can differ slightly from a run without the cache (same quality:
                     bench/results/2026-09-27-cache-parity).
                     PROFILE, ranked by routing frequency, no eviction.
    strata generate: pre-filled 2513 of 2513 slots from the profile in 0.5 s (10264 MB/s); slot 0 verified
    strata generate: R4 hit path ON - resident experts are computed on the GPU
    strata generate: 5 expert-pool workers + the host thread
    strata generate: session is up (engine 0.1.64)
    strata generate: token graph hit path: 2513 resident experts, decided on the device
    strata: prompt kernels: chunks of 4608 tokens or more run 48 of 48 layers on the fused kernels; shorter chunks run 48 on the staged fused entry.
    strata serve: small expert cache: the default prompt path would read in chunks of 7680 tokens (under 8192), so prompts are read the way that suits a card with little memory and, from two chunks, the layer-major walk (chunk 7168 tokens); STRATA_PF_SMALL=0 keeps the default path, STRATA_PF_FUSED / STRATA_PREFILL_LAYER_MAJOR set by hand win
    strata serve: prompt chunk auto: 7168 tokens, a 384-slot ring
    strata serve: the prompt path borrows 2223 CUDA0 cache slots (4.26 GiB)
    strata hc: CUDA0: the hyper-connection read runs as staged (the norm per token and stream, the down projection's activations staged ahead by cp.async); checked bit for bit against the plain read on this card (STRATA_HC_SPLIT=1 or 0 for the earlier ones)
    strata verify: window up to 12 tokens, 109.5 MiB of device buffers
    strata mtp: draft head over 40525 tokens (81.2 MiB)
    strata serve: conversation cache: SSD tier in <data>/conversation-spill/1943139-352dc825b3df109e, up to 32768 MiB
    strata serve: PCIe share: start 0.30 from 5 pool workers (assumed 20 GB/s of expert bytes) and the 11.1 GB/s link (was 0.31)
    strata serve: adaptive overlap: off: not with STRATA_ADAPT_LAG / STRATA_ADAPT_NOWAIT
    strata serve: the adaptive tier's evictions reach the device's table before the next window (STRATA_ADAPT_EVICT_SYNC=0: when its copies land)
    strata serve: adaptive tier: blocking, and the asynchronous tier is tried once the session shows the swaps hold the windows up (--adapt-async 0 or STRATA_ADAPT_ASYNC=0: never)
    strata serve: 680 MiB of VRAM free with everything loaded
    strata verify: capturing the 1-token window (680 MiB of VRAM free)
    strata verify: captured the 1-token window (upload no error, sync no error) in 31.5 ms
    strata verify: capturing the 2-token window (666 MiB of VRAM free)
    strata verify: captured the 2-token window (upload no error, sync no error) in 21.1 ms
    strata verify: capturing the 3-token window (652 MiB of VRAM free)
    strata verify: captured the 3-token window (upload no error, sync no error) in 19.0 ms
    strata verify: capturing the 4-token window (638 MiB of VRAM free)
    strata verify: captured the 4-token window (upload no error, sync no error) in 19.4 ms
    strata verify: capturing the 5-token window (622 MiB of VRAM free)
    strata verify: captured the 5-token window (upload no error, sync no error) in 17.0 ms
    strata verify: capturing the 6-token window (608 MiB of VRAM free)
    strata verify: captured the 6-token window (upload no error, sync no error) in 17.1 ms
    strata verify: capturing the 12-token window (594 MiB of VRAM free)
    strata verify: captured the 12-token window (upload no error, sync no error) in 40.4 ms
    strata serve: window warm-up: ms per width 1:79 2:111 3:132 4:165 5:201 6:240 12:442, total 1537 ms
    strata serve: --lookup-max-t 12: MTP windows up to 4 tokens, window sizes 1,2,3,4,5,6,12
    

    Prepared with AI assistance; all numbers are from my own runs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions