fix: skip MTP speculative decoding when no mmproj available (#2451)#2458
fix: skip MTP speculative decoding when no mmproj available (#2451)#2458bong-water-water-bong wants to merge 1 commit into
Conversation
…ade-sdk#2451) MTP speculative decoding in llama.cpp requires multimodal support (glm4-moe.cpp accesses vision tensors during draft generation). Models with MTP heads but no multimodal projector (e.g. quantized GLM-4.5-Air GGUF) crash at first prompt with 'This GGUF does not support multimodal'. Fix: check that an mmproj file is present before enabling --spec-type draft-mtp. When MTP is detected but no mmproj is found, log a warning and skip speculative decoding. The model loads and runs normally without MTP acceleration.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
github-actions
Bot
added
engine::llamacpp
fl0rianr
left a comment
fl0rianr
left a comment
There was a problem hiding this comment.
Blocking this: MTP does not imply mmproj. mmproj_path.empty() is a good signal for the reported GLM-4.5-Air crash, but it is not a valid global condition for enabling/disabling draft-mtp. There are MTP models without it.
Please narrow the guard to the crashing architecture/path, e.g. GLM4-MOE without mmproj/hf_load, or otherwise detect the exact llama.cpp condition that leads to ubatch.embd && !use_mrope. Text-only MTP models should not lose speculative decoding just because they do not ship a multimodal projector.
2 participants
Summary
Fixes #2451 — GLM-4.5-Air-UD-Q4K-XL-GGUF crashes on first prompt with `glm4-moe.cpp:149: This GGUF does not support multimodal.`
Root Cause
The GLM-4.5-Air GGUF has MTP (Multi-Token Prediction) tensor heads, which Lemonade detects via GGUFCapabilities and adds the `"mtp"` label at runtime. This causes `llamacpp_server.cpp` to pass `--spec-type draft-mtp` to llama-server.
However, the quantized GGUF only contains MTP head tensors — it does not include the multimodal projector (mmproj). When llama.cpp's glm4-moe handler tries to access vision tensors during MTP draft generation, it hits a hard abort at `glm4-moe.cpp:149`.
Fix
In `llamacpp_server.cpp`, only enable `--spec-type draft-mtp` when an mmproj file is available (or when `hf_load` is used, which lets llama-server resolve it from the HF repo). Without an mmproj, skip MTP speculative decoding with a warning — the model loads and runs normally, just without MTP acceleration.
Testing
Build from the fix branch, load GLM-4.5-Air-UD-Q4K-XL-GGUF, and send a prompt. The model should respond correctly instead of crashing the llama-server process.