Repository navigation
Description
Measurements from one Strix Halo Windows machine, run against engine 0.1.48
(Control 0.2.3). This continues the Windows deployment notes in #3029 and the
Strix Halo discussion in #3032; every number below is from this machine and each
A/B was interleaved (A/B/A/B, fresh process per arm, median), because same-session
drift here reached 3-5%.
Three findings that may matter upstream:
STRATA_PF_FUSED_KQ=1+STRATA_PF_F16=1is worth +12.3% prefill on a
6.4K-token prompt on IQ3_XXS (854.3 against 760.4 tok/s, paired +12.5/+10.5/
+12.0 across three starts), and flat at 3.2K. IQ3_S gives +8.0%. It is the
largest untapped lever I found on this card.- The shipped hipBLASLt table is already at this machine's optimum: retuning
30 shapes picked the same solution on 22, and the engine A/B of the retuned
table is 629.0 against 629.7 tok/s. The twobf16 N=1rows crash
tune_hipblasltwith 0xC0000005 (M=1 matvec), so those rows cannot be
recalibrated at all. - The remaining gap is the bf16 dense GEMM:
strata_wmma_gemm_bf16is 2.45-2.6x
slower than hipBLASLt on the two dominant prefill shapes (22.42 against
8.76 ms for 4096x10240x2560), andSTRATA_WMMA_GEMM=1costs 2.4% overall.
Also in the document, with measurements: DFlash2 unlocked under --serve (a
one-line gate change) is -9.5% prefill and -1.7% decode at cap 5, -15.8% at cap 8,
with draft acceptance identical to MTP; the PLE table is already IQ4_NL (90 B/row)
and --ple-io mmap/ram change nothing (+0.9% / -5.0%); engine 0.1.47 -> 0.1.48
is parity on a configuration that already sets STRATA_HC_Q8, the tuning table and
--prefill 16384.
gfx1151 on Windows: measurements against engine 0.1.48
Notes from tuning one Strix Halo machine. Every number here was measured on this
machine; where a claim comes from the release notes it says so. The point of the
document is to save the next person the dead ends, and to hand the author a
reproduction path for the two things that still look open.
0. Machine, model, method
- Ryzen AI Max+ 395 / Radeon 8060S (gfx1151), 128 GB LPDDR5X, Windows 11 x64,
ROCm 10.2.0a20260930, engine 0.1.48 (windows-x86_64-hip-rdna3package),
Control 0.2.3, started throughserve/server.py(strata.exe --serve). - Models:
swift-abl-1.5-iq3_xxs(IQ3_XXS, 77.87 GB) and IQ3_S; config as in
STRIX_HALO_WINDOWS.mdsection 3 plusSTRATA_PF_FUSED_KQ=1and
STRATA_PF_F16=1(section 2 below). - Protocol: interleaved A/B (A/B/A/B/A/B), one fresh engine process per arm, one
warm-up request, 3-4 measured requests per process, median. Prompts are one
Chinese text repeated to 1,646 / 3,218 / 6,362 tokens; temperature 0; a unique
nonce per request. Same-session drift reached 3-5%, so only paired numbers are
used. Prefill isprompt_per_second, decodepredicted_per_second.
1. The shipped hipBLASLt table has no headroom here
tools/hip/tune_hipblaslt over the shipped 32-row table, 3 passes, per-solution
median of the three:
- 22 of the 30 non-degenerate rows pick the same solution as the shipped table.
- The 8 rows that differ look dramatic in a single pass (f16 512x2560 T=8192:
solution 2542 measures 13-52 ms against 0.7-1.6 ms for 2541) but the same
solution is 0.4-1.0 ms in another pass. The timings are bimodal, not stable. - Engine A/B, shipped table against the retuned one: prefill 629.0 against 629.7
tok/s, decode 47.5 against 46.8. No change.
Reproduce: tune_hipblaslt <cases> --tuning-out table.txt, then diff the two
tables and A/B them with STRATA_HIPBLASLT_TUNING.
Bug found on the way: the two bf16 N=1 rows (bf16 1 2560 1 4096/8192) crash
the tuner with 0xC0000005 (access violation inside hipBLASLt) for the M=1 matvec
shape. The tuner cannot recalibrate those rows; the shipped values are all a
rebuild can have. A guard that skips N=1 with a note would be better than a
crash.
2. STRATA_PF_F16 + STRATA_PF_FUSED_KQ is a real long-prompt win
Opt-in today. Measured on IQ3_XXS, 6,362-token prompt, three paired starts:
| arm | prefill tok/s | paired differences |
|---|---|---|
| baseline | 760.4 | - |
STRATA_PF_FUSED_KQ=1 |
773.6 (+1.7%) | -2.2 / -0.8 / +3.3 % |
+ STRATA_PF_F16=1 |
854.3 (+12.3%) | +12.5 / +10.5 / +12.0 % |
At 3,218 tokens the same arms are flat (618.5 against 621.5). On IQ3_S the two
switches give +8.0% (893.0 -> 964.7 tok/s at 6,362 tokens). Decode is unchanged.
Output text stays coherent; a fixed prompt at temperature 0 gave 334 against 329
characters with the same argument structure, i.e. wording-level differences, as
the rounding note says.
The path runs for IQ3_XXS (no "off"/fallback line in the log), although
STRIX_HALO.md section 5 describes the switch with IQ3_S / IQ4_NL numbers.
Suggestion: document this pair as the long-prompt setting on gfx1151 (it is
what --prefill chunks above ~4K tokens need), or enable it automatically for
T >= 4096 behind the existing KL gate. Today a reader has to find two switches
and two conditions; the gain is larger than anything else left in the table.
3. DFlash2 under --serve: possible, not profitable
The gate at src/program/generate.cpp:5478 rejects o.serve. Removing that one
condition lets DFlash2 serve (log: windows capped at 5 tokens, no fallback).
Measured with the same local 0.1.48 build in both arms:
| arm | prefill tok/s | decode tok/s |
|---|---|---|
| MTP | 880.4 | 55.0 |
| DFlash2 cap 5 | 792.9 (-9.9%) | 51.7 (-1.7%) |
| DFlash2 cap 8 | 792.9 (-9.9%) | 46.3 (-15.8%) |
Draft acceptance is the same in both arms (MTP 25-27 of 33-38, DFlash2 25-27 of
33-38), so the drafter's quality is not the problem: the loss is per-round
drafting cost (CLI: 9.49 ms/round against MTP 8.06) plus the tap write of
df2_tap_rows for every layer of every chunk (the prefill 9-10%) plus 420 MiB
of VRAM. Widening to 8 makes it worse, and kVerifyMaxT = 8 caps the window
anyway; the "wider verify" hypothesis is not supported on this model.
Suggestion: keep the gate, but the release note could say that serve is
excluded because of the taps' cost, not because DFlash2 is untested.
4. The PLE table is already 4-bit; its I/O is not the bottleneck
include/strata/ngram/ple_reader.hpp says the table is 320,001,536 rows of 90 B
(IQ4_NL, 26.8 GiB); that is what this IQ3_XXS shard is. There is no q8 -> q4
gain to take (hipfire's +14% is against its own q8 PLE).
--ple-io A/B at 6,362 tokens, three starts per arm:
| mode | prefill tok/s |
|---|---|
direct (default) |
863.5 |
mmap |
871.2 (+0.9%) |
ram (mlock 26.8 GB) |
820.4 (-5.0%) |
The default's unbuffered reads plus the 1M-row cache, page dedup and sorted
submission are already off the critical path; locking the table in RAM only
steals page cache from the experts. No change recommended.
5. The real gap is the bf16 dense GEMM
Standalone benchmark (same buffers, same GPU, rel_l2 2e-6 against hipBLAS):
| shape, bf16 | strata_wmma_gemm_bf16 |
hipBLASLt best | ratio |
|---|---|---|---|
| T=4096 N=10240 K=2560 | 22.42 ms (9.6 TFLOP/s) | 8.76 ms (24.5 TFLOP/s) | 2.6x slower |
| T=4096 N=2560 K=2560 | 5.48 ms (9.8 TFLOP/s) | 2.24 ms (24.0 TFLOP/s) | 2.45x slower |
The numbers are the tuned-table medians for the same shapes (the 64x64 WMMA
kernel versus hipBLASLt solution 1254 on the first shape). In the engine,
STRATA_WMMA_GEMM=1 costs 2.4% prefill (867.6 -> 846.6) and 1.6% decode, i.e.
the generic bf16 WMMA path is not a substitute for the library either.
This is where the remaining speed is. Prefill at 780-880 tok/s against a 250 GB/s
machine that can do 24.5 TFLOP/s in the library; another engine on the same
hardware reports 2,100-2,400 tok/s with 4-bit weights, which implies roughly a
2.4x better dense GEMM plus the smaller weight stream. Matching hipBLASLt is not
enough (it is the current path); a bf16 tile kernel would have to be ~2.5-3x
faster than the existing one to matter.
Question for the author: is a 128x256 (or 256x128) bf16 tile kernel with
split-K and double-buffered LDS on the roadmap, or is hipBLASLt considered the
ceiling for bf16 on gfx11? The 64x64 kernel is 2.5x off, and the measurement
above is a ready-made target.
6. Engine 0.1.47 -> 0.1.48 on this configuration: parity
Three paired starts, 6,362-token prompt: prefill 877.3 against 874.3 (-0.3%),
decode 51.8 against 51.5 (-0.6%). Expected: this config already sets
STRATA_HC_Q8=1 (the 0.2.3 default), already points
STRATA_HIPBLASLT_TUNING at the table, and passes --prefill 16384 rather than
auto, so none of the three 0.2.3 Strix Halo changes apply. The release note's
+6-19% / +5-14% figures are against a 0.2.2 default configuration.
7. Local changes used for these measurements
strata-swift-abl.jsonandstrata-iq3_s.json: added
"STRATA_PF_FUSED_KQ": "1","STRATA_PF_F16": "1". Kept.src/program/generate.cpp: removedo.servefrom the DFlash2 gate for the
section 3 A/B, then reverted. Not proposed as a patch as it stands.- Built
tune_hipblasltand a smallbench_gemm_shape(links
src/prefill/wmma_gemm.cu+ hipBLAS, same allocation, event timing). - The locally built 0.1.48 engine needs the packaged DLL closure to start
(0xC0000139 otherwise); the package'srocm/binnext to the exe is what makes
it load.
8. Summary
| lever | measured effect | keep? |
|---|---|---|
| 18 gfx1151 defaults off | -6.2% prefill, -17% decode | keep on |
STRATA_SH_STREAM=0 on Windows |
+13% decode (earlier doc) | already set |
| hipBLASLt table retune | +0.1% (noise) | no change |
STRATA_PF_FUSED_KQ + STRATA_PF_F16 |
+12.3% prefill at 6.4K, 0 at 3.2K | yes (opt-in) |
| DFlash2 in serve (patched) | -9.5% prefill, -1.7% decode | no |
PLE --ple-io mmap/ram |
+0.9% / -5.0% | no change |
bf16 WMMA GEMM (STRATA_WMMA_GEMM=1) |
-2.4% prefill | no |
Open: a bf16 dense GEMM that beats hipBLASLt (section 5), and the tuner's N=1
crash (section 1).
Activity
github-actions commentedon Oct 23, 2026
Thanks for the issue. To keep the tracker readable, every item needs an area prefix in the title and the parts of the issue form filled in. This one is missing:
Edit the issue (title and description) and this note goes away on its own. If it is still missing parts in 5 days it will be closed, which is not a rejection: fix it and comment
/reopen.Details: CONTRIBUTING.md
mistral13 commentedon Oct 23, 2026
Are the improvements from STRATA_PF_FUSED_KQ=1 and STRATA_PF_F16=1 also present in unsloth-ud-iq4_xs?
Do you also observe that only a single CPU core is being utilized during decoding, or is this behavior specific to unsloth-ud models?
I'm also interested in finding the optimal parameters for this model.
I haven't tested version 0.2.3 yet, but in earlier versions, using --mmap-experts together with STRATA_SH_STREAM=0 resulted in a decoding performance improvement of more than 10%. Have you tested this combination yet?
[-][gfx1151] Windows measurements on engine 0.1.48: PF_F16 +12% prefill, the tuning table at its local optimum, and the bf16 GEMM gap[/-][+][Engine] gfx1151 Windows measurements on engine 0.1.48: PF_F16 +12% prefill, the tuning table at its local optimum, and the bf16 GEMM gap[/+]github-actions commentedon Oct 24, 2026
Thanks, this follows the format now.
HaiYunChuang commentedon Oct 24, 2026
On UD-IQ4_XS: I have not run that model, so I cannot confirm the gain. The F16 prompt-expert path covers IQ4_XS gate/up (
native_f16_kernel) and IQ4_NL down inmoe_fused_iq.cu, but UD mixes tensor types, so the only reliable check is the log plus a long-prompt A/B: withSTRATA_PF_F16=1there should be no fallback line, and a 6.4K-token prompt should be clearly faster than the same run withSTRATA_PF_FUSED_KQ=0 STRATA_PF_F16=0. On IQ3_XXS the gain is 0 at 3.2K and +12% at 6.4K, so measure a long chunk, not a short one.On one CPU core during decode: in our runs the expert cache holds all 24,576 slots and the profile line reads
CPU experts 0.00 distinct / 0.00 routed per layer, so no expert work reaches the CPU and the pool stays idle; the single busy core there is the host thread. If your line shows CPU experts above zero (fewer slots than experts, or a small VRAM budget), then--pool-tasks/--pool-workersare the knobs, and it is not model-specific.The configuration behind the numbers in this issue:
--mmap-experts --expert-cache auto --prefill 16384 --spec 4 --spec-min-p 0.5 --lookup-chain 3 --mtp-q4 all --mtp-window 8192 --kv int8 --max-context 262144, withSTRATA_PF_FUSED=1 STRATA_PF_FUSED_KQ=1 STRATA_PF_F16=1 STRATA_PF_GEMM=1 STRATA_HC_UPMIX=1 STRATA_PA_FAST=1 STRATA_HIP_WMMA=1 STRATA_SELECT_WMMA=1 STRATA_HC_Q8=1 STRATA_PF_SWITCH_MIN_T=4096 STRATA_PREFILL_STREAM_MIN=128 STRATA_SH_STREAM=0and the shipped gfx1151 hipBLASLt table. On Windows,STRATA_SH_STREAM=0is the one inverted default (+13% decode here).HaiYunChuang commentedon Oct 27, 2026
Update on 0.2.7 / engine 0.1.52: the new PLE residency policy costs 15-16% prefill on this machine, and one switch removes it
Same machine as section 0 (Ryzen AI Max+ 395, 128 GB, Windows 11, ROCm 10.2.0a20260930), source v0.2.7 started through
serve/server.py,swift-abl-1.5-iq3_xxs(IQ3_XXS) with the section 2 config (STRATA_PF_FUSED_KQ=1,STRATA_PF_F16=1,STRATA_SH_STREAM=0,--prefill 16384,--mmap-experts). Prompt 6,362 tokens, interleaved A/B, one fresh engine process per arm, warm-up plus 3 measured requests, median. Engine 0.1.50 (0.2.6 package) against engine 0.1.52 (0.2.7 package), both under the 0.2.7 server.STRATA_PLE_AUTO_RAM=0Round 3 paired deltas: -1.3% / -0.1% / -0.1%.
Cause. Every 0.1.52 start logs:
and only in the 0.1.52 arms:
So
ple_residency.hppadmits the table (26.8 GiB + arena + 8 GiB headroom fits in 128 GB) and locks it on every start. That is the mode measured in section 4 at -5.0% (--ple-io direct863.5 against--ple-io ram820.4). On this machine the direct reads plus the 1M-row cache were already off the critical path, and the locked table takes the RAM that the mmap'd experts read through the page cache; the conversation cache is then dropped as well. Net -15 to -16% prefill.Suggestion. On Windows gfx1151 (unified memory) the "table + arena + 8 GiB" test is not the right admission rule: the lock's cost is paid from the same pool the mmap'd experts read through. Either leave the policy off for gfx1151/Windows by default, or require the table to be a small fraction of available RAM.
STRATA_PLE_AUTO_RAM=0restores the direct reads and gives parity here, but a reader who does not know the switch loses 15%.Nothing else regresses: with the switch set, 0.1.52 is within -0.9% prefill of 0.1.50, and decode is inside this machine's same-session drift (0.1.50 51.6-58.1, 0.1.52 46.0-56.8 across the three rounds; some requests ended at 28/29 completion tokens, so the decode medians are not clean).
Niko1221 commentedon Oct 29, 2026
Thank you for the interleaved numbers: 874.9 / 869.4 against 730.3 / 737.1 tok/s, and 878.9 / 870.7 with
STRATA_PLE_AUTO_RAM=0, is a clear result. We read the residency policy (ple_residency.hpp) and agree with your cause: it has no check for a GPU that shares the PC's memory (Strix Halo) or for a Windows HIP build, and it locks the table out of the pool the memory-mapped experts read through. We have not changed the default. Until thenSTRATA_PLE_AUTO_RAM=0(or--ple-io direct) is the setting to use on your machine.Niko1221 commentedon Nov 5, 2026
Thanks for taking the time to report this.
Strata has changed a lot since this was opened, and 0.2.11 is out now, so many older reports no longer apply. We're closing issues older than a week that have had no new reports in the last week. Closed here does not mean rejected.
To reopen it takes about two minutes:
/reopen. A check runs and either reopens the issue or lists what is still missing.The rules are in CONTRIBUTING.md.
HaiYunChuang commentedon Nov 9, 2026
Update (Nov 3–8, same box): engine 0.1.52 → 0.1.58, the BIOS carve-out finding, and a DFlash2 negative result on a Swift fine-tune
Same hardware (Ryzen AI Max+ 395, 128 GB, Windows 11, gfx1151). All numbers are interleaved independent-process A/B (arms' logs checked identical each round), temperature 0, the 6.4K prompt from my earlier runs, median of 3 per start. The four guards stayed in place throughout (
STRATA_PF_FUSED_KQ=1,STRATA_PF_F16=1,STRATA_SH_STREAM=0,STRATA_PLE_AUTO_RAM=0).1. Engine upgrades
SMALL_UNR,GR_Q8_PFS, 18 → 20), and that is all an i-quant set picks up.2. The BIOS UMA carve-out was worth +27–29% prefill — possibly the largest single knob on a Strix Halo Windows box
This box shipped with UMA Frame Buffer Size = 32 GB. Your test machine runs 64 GB ("128 GB (64 GB set aside for graphics)", 0.2.10 notes). With 32 GB the IQ3_S expert cache (46.84 GiB) overflows ~15 GB into WDDM-managed memory:
One BIOS setting: +27% / +29% prefill, and most of the gap to your published 1,424 @8k closes (the remaining −8% is plausibly the Swift fine-tune and variant differences). Cost: Windows sees 63.6 GB, so the conversation cache stays off. This may deserve a line in the Strix Halo guide — a user on the default 32 GB carve-out silently loses ~25% prefill and ends up reporting "much slower than advertised."
3. DFlash2 tuned pack on a Swift fine-tune: negative — and the exclusion is exactly right
Swift-abl IQ3_S (SC117's abliterated Swift 1.5) is my main set, so
model_namedoesn't match_SUPPORTED_NAMEand serve correctly fell back to MTP ("DFlash2 is for Qwen3.8-Flash-Next only"). I ran it anyway by naming the configqwen3.8-flash-next-iq3_s, which appliesSET_DEFAULTS[("hip","iq3_s")]: the tuned selq4 pack (sha256-verified against the pin), conf 0.35,--spec 6(windows capped at 5 by the engine). One setup note: the serve side finds the drafter viadata_dir_of, which walks two levels up from--pack— i.e. the model data folder, not the install'sdata\directory where the stock pack lives.--spec 4)All three DFlash2 reps landed below all three MTP reps. Draft acceptance was only 50–58%: the drafter is trained on the stock distribution, and Swift's shorter thinking plus the abliteration shift costs more in verification than the accepted drafts return. Conclusion: Swift sets stay on the MTP — the
SUPPORTED_FAMILIESexclusion is right, and this fills in the one number behind it. The tuned pack stays in the model data folder; the stock ISTA IQ3_S config matches by name and can use it any time.Happy to run anything specific on this box — the interleaved A/B harness is reusable.
HaiYunChuang commentedon Nov 22, 2026
Update (Nov 22): 0.2.16's VRAM regression is fixed in 0.1.65 — but the prompt path on gfx1151 + IQ3_S is now 23% slower than 0.1.60, so this box rolled back again
Same hardware and method as before (Ryzen AI Max+ 395, 128 GB, Windows 11, 64 GB UMA carve-out, swift-abl IQ3_S, the 6.4K/8K prompts, medians).
1. The good news first: the 0.1.62 memory regression is gone. With 0.1.65 the engine loads to 45,247 MiB of VRAM free — the same figure as 0.1.60 (0.1.62 left only 24,969; it also held ~20 GB more with everything loaded). Thank you for the quiet fix.
2. The prompt path, however, regressed on this configuration. 0.2.14+ moved the prompt experts to Strata's own kernels, and on 0.1.60 the old guard pair still won:
STRATA_PF_F16=1+STRATA_PF_FUSED_KQ=1was 5.5% faster than the new path (my Nov 16 numbers here: 1,332 vs 1,260 @6.4K). On 0.1.65 that old path no longer engages — the log now says "prompt experts on the fused int8 kernels" with the same config that used to take the F16 route — and nothing I tried recovers the speed:native_h11p_kernelroute)So the ceiling on 0.1.65 is ~1,015 tok/s — 23% below 0.1.60. Decode is unchanged (44–48 throughout). The release notes' RDNA3 speedups are all NVIDIA figures; on gfx1151 with an i-quant set the new kernels have not caught up with the old F16 fused route yet.
3. What this box does now: stays on v0.2.15 / 0.1.60 (second rollback; the first was 0.2.16 for the VRAM issue). The retest trigger I keep is simple — when a release looks relevant, measure
PF_F16=2once; if it clears ~1,300 @6.4K, the new path has arrived and the guards can retire.Happy to run any specific build or setting you want measured on this machine — the interleaved harness takes minutes per arm.