[Blog] Add Humming integration post - #380
Conversation
BBuf
left a comment
BBuf
left a comment
There was a problem hiding this comment.
A few concise suggestions on terminology, reproducibility, and figure presentation.
| For unquantized FP16/BF16 checkpoints, an environment variable can specify the target weight format. The following example converts compatible linear and MoE weights into group-wise INT4 during model loading: | ||
|
|
||
| ```bash | ||
| export SGLANG_HUMMING_ONLINE_QUANT_CONFIG='{"dtype":"int4","group_size":128}' |
There was a problem hiding this comment.
Could we show a complete sglang serve ... --quantization humming example? These variables configure Humming but do not select the backend by themselves.
|
|
||
| #### Kimi-K2.6: Text Input | ||
|
|
||
| The text-input latency evaluation compares Marlin WINT4A16, Humming WINT4A16, and Humming WINT4AFP8 on Kimi-K2.6 under TP8, using group-size-32 INT4 expert weights. |
There was a problem hiding this comment.
The strongest comparison also changes activation precision from A16 to A8. Could we separate backend gains from quantization gains and include accuracy/quality results?
| <em>Figure 5. Kimi-K2.6 Agentic Pareto curve across concurrency levels 1, 2, 4, and 8.</em> | ||
| </p> | ||
|
|
||
| At concurrency levels 2, 4, and 8, both Humming WINT4A16 and WINT4AFP8 improve interactivity and per-GPU throughput over Marlin WINT4A16. Humming WINT4AFP8 reaches its knee at C=4 with 35.19 tok/s/user and 3,928.20 tok/s/GPU, improving the two metrics by 9.35% and 18.06%, respectively. |
There was a problem hiding this comment.
Could we define the knee-selection rule and mark C=4 in Figure 5? Otherwise “reaches its knee” is subjective.
| <em>Figure 6. DeepSeek-V4-Flash Agentic Pareto curve across concurrency levels 1, 2, 4, and 8.</em> | ||
| </p> | ||
|
|
||
| At every tested concurrency, both Humming configurations move the Pareto curve above and to the right of Marlin MXFP4A16. Humming MXFP4A16 improves interactivity by 5.79%–13.54% and per-GPU throughput by 8.65%–14.46%. Humming MXFP4AFP8 extends the interactivity gains to 11.04%–21.74% and the per-GPU throughput gains to 14.91%–26.22%. |
There was a problem hiding this comment.
Figure 6 places Humming W4A16 at C=1 to the left of Marlin, so “at every tested concurrency” and the positive 5.79% lower bound cannot both be correct. Please verify this claim.
cd4fd6c to
ae04694
Compare
ae04694 to
13c24d6
Compare
Thanks for the review! Addressed in the latest update. |
2 participants
Summary
This PR adds a new LMSYS blog post introducing Humming and its integration into SGLang.