Context Shift with Quantkv on Gemma-4

[Abdulhanan535]

built latest changes, was using Gemma 4 31B, context shift isn't supported when using quantkv 1, koboldcpp simply crashes.

Gemma 3 27B works fine.

[LostRuins]

Hmnm yeah, that seems to be the case. Upstream forces SWA for this model, so context shifting is never triggered. I suspect its simply not implemented. Shifting does work fine if KV is f16...

[Abdulhanan535]

Hmnm yeah, that seems to be the case. Upstream forces SWA for this model, so context shifting is never triggered. I suspect its simply not implemented. Shifting does work fine if KV is f16...

yep, but 8k fp16 kv takes 6gb vram. :\

[LostRuins]

Did you enable SWA with flash attention? That drastically reduces KV memory usage.

[Abdulhanan535]

Did you enable SWA with flash attention? That drastically reduces KV memory usage.

idk how lol.
my command:
./koboldcpp_linux model.gguf --usecublas mmq --gpulayers $Layers --contextsize $ContextSize --tensor_split 16 16 --port 2222 --quiet --flashattention

[Abdulhanan535]

[Abdulhanan535]

Did you enable SWA with flash attention? That drastically reduces KV memory usage.

it does reduce vram usage to 1gb but no context shift kills me.

[LostRuins]

Yeah thats the tradeoff. Enable with --useswa

[Abdulhanan535]

@LostRuins one bug I found when using context sift with gemma4, sometimes it doesn't shift the context and reprocesses the whole context. like 1/3 chance.

./koboldcpp_linux model.gguf --usecublas rowsplit mmq --gpulayers 99--contextsize 8192 --port 2222 --quiet --flashattention

[Phil209]

@Abdulhanan535 I noticed the same~1/3 chance, and with Qwen3.5 aswell, because about every third time there's a long delay as it reprocesses the context.

Also, something is fundamentally wrong with Gemma 4's attention/context. When I try the same prompts with the latest llama.cpp build it does much better than in Koboldcpp so they may have made some important changes.

Since the responses are off I asked what I just wrote and it was garbled. Maybe one or two words were correct. So it's responding inappropriately because it can't accurately see the user's prompt.

[Abdulhanan535]

@Abdulhanan535 I noticed the same~1/3 chance, and with Qwen3.5 aswell, because about every third time there's a long delay as it reprocesses the context.

Also, something is fundamentally wrong with Gemma 4's attention/context. When I try the same prompts with the latest llama.cpp build it does much better than in Koboldcpp so they may have made some important changes.

Since the responses are off I asked what I just wrote and it was garbled. Maybe one or two words were correct. So it's responding inappropriately because it can't accurately see the user's prompt.

For me responses are same but the context shift issue is still there. I built kcpp again after the tokenizer got fixed and pulled into kcpp. I guess we've to wait for fixes.

[henk717]

Qwen3.5 is not relevant, it is impossible to use context shift with an RNN model so context shift is not used there at all.
Coherency wise were aware of some issues, including the fact most Gemma4 quants are incorrect which we will add a workaround for in 1.111.1.

Context Shift / fast forward can only trigger if nothing new was added to the context of course. So do make sure this is the case and your UI isn't flipping back and forth, uses world info, embedding, etc. If you do have those scenarios you may need to combine it with SmartCache.

[Phil209]

@henk717 I don't want to take this thread about Quantkv off topic so I'm going to delete this comment later today. Just thought I'd bring to your attention what you're already aware of (a disconnect between what the latest llamacpp and kobodcpp 1.111 is outputing).

The pattern I'm seeing is it's regularly taking everything extremely literal and making the outputs extremely structured. So for example, if you use 9 vs nine (see below) it will do weird things, or if you ask for things like "the 6 main cast members" it will return things like 'I didn't find any mention of "6 main cast members" from that movie'.

I tried the default settings and chat template, various modifications, different models (this on was from Bartwoski's Q4_K_M of 26B-a4b).

"Make a simple list of 9 single word synonyms for extra." -Response: "1.2.3.4.5.6.7.8.9."

[LostRuins]

We've noticed some issues pertaining to the chat template and BOS token, a patch release is in the works and will be updated shortly, for now please hang tight

[Abdulhanan535]

Qwen3.5 is not relevant, it is impossible to use context shift with an RNN model so context shift is not used there at all. Coherency wise were aware of some issues, including the fact most Gemma4 quants are incorrect which we will add a workaround for in 1.111.1.

Context Shift / fast forward can only trigger if nothing new was added to the context of course. So do make sure this is the case and your UI isn't flipping back and forth, uses world info, embedding, etc. If you do have those scenarios you may need to combine it with SmartCache.

But --useswa causes memory issues doesn't it? gemma-4