gemma-4-12B-it: Infinite thinking loop, system prompt non-adherence (>10k tokens), and context loss in multi-turn conversations

[titansonico10]

Bug Report + Improvement Request

Model

google/gemma-4-12B-it (original, non-fine-tuned)

Runtime

llama.cpp / LM Studio (local inference)

---

Issue 1 — Infinite thinking loop (no exit condition)

The model frequently enters an infinite loop inside the <|channel>thought ... <channel|> reasoning block and never terminates generation. The only way to recover is to manually stop generation.

This has been independently confirmed by multiple users running the model locally via llama.cpp and Ollama:

• https://discuss.ai.google.dev/t/gemma4-how-to-prevent-looping-output-in-thinking/173205

Partial workaround found by the community: increasing repeat-penalty to 1.08 with repeat-last-n 4096 reduces — but does not eliminate — the issue.

---

Issue 2 — Complex system prompts (>10,000 tokens) cause unreliable behavior

When a system prompt exceeds ~10,000 tokens, the model exhibits several failure modes:

• Extremely long time spent "reading" the system prompt before producing any output
• Contradicts or ignores rules explicitly stated in the system prompt
• Does not know what to do next after processing the instructions, producing confused or empty responses
• Mixes up steps, skips required actions, or applies rules from one context to another incorrectly

This makes the model unreliable for complex agentic workflows, roleplay frameworks with detailed character cards, or any use case requiring strict instruction-following over a long system prompt.

---

Issue 3 — Conversation context loss in multi-turn sessions

In longer conversations, the model progressively loses track of:

• Instructions given in the system prompt (behaves as if it never read them after a few turns)
• Content discussed in earlier turns of the conversation
• The current state or "thread" of the dialogue

This is distinct from simply running out of context window — it appears to be a failure of attention over the full sequence even when the content is still within the model's context length.

---

Expected Behavior

• The thinking channel should always terminate and produce a coherent final response
• Rules and steps defined in the system prompt should be followed consistently regardless of prompt length
• The model should maintain coherent reference to prior conversation turns throughout the session

Suggested Improvements

• Add a maximum thinking budget / token limit for the reasoning channel to prevent runaway loops
• Improve instruction following for system prompts in the 5,000–15,000 token range
• Investigate attention degradation over long multi-turn contexts and consider fine-tuning on longer conversation datasets

---

Environment

• Model: google/gemma-4-12B-it
• Backend: llama.cpp (also reproducible with Ollama)
• Hardware: Local GPU inference
• Quantization: Q4 to Q8 variants (issue present across quantization levels)

[soenning-ai]

Some points that i figured out working well:

If you log thinking, check it. My thinking loops without an end results on failed tooling for within thinking tool calls. Another point is reread fails and get worse after every retry within one thinking round. I have around 6k limit thinking tokens and set reasoning to high.

I am working with 16k system prompts and that's works totally fine, gemma 4 wrote it's own instructions in my setup. The Prompt has a warm and calming tone.

[titansonico10]

Update — PR #732 does not resolve this issue.

@BhadaneAaditya opened PR #732 claiming to close this issue. A technical review of the changes:

• Issue 1 (infinite loop): Fixed correctly via max_thinking_tokens token budget enforcement. Deterministic and effective.
• Issue 2 (system prompt degradation): Not fixed. The PR adds a UserWarning at 8,192 tokens with no enforcement, no truncation, and no attention-level change. The sliding window architecture problem is untouched.
• Issue 3 (multi-turn context loss): Not fixed. A turn counter warning every 10 turns with no actual KV-cache refresh or attention re-prioritization does nothing at the model level.

Issues 2 and 3 stem from Gemma 4's architecture (limited global attention layers, sliding window degradation) and cannot be patched at the Python wrapper level without model-level changes or fine-tuning. This issue should remain open.

[titansonico10]

Additional failure mode: empty thinking block — channel opens but generates zero reasoning content

Running google/gemma-4-12B-it (Q8_0 GGUF) locally via llama.cpp / Jan 0.8.4 on Windows (Vulkan backend), a third thinking failure mode has been observed alongside the infinite loop described above:

Observed output:

Thought for 1 seconds
<|channel>thought
Done

The model:

1. Opens the thinking channel correctly
2. Emits <|channel>thought as a literal token rather than generating reasoning content
3. Closes the block immediately with zero reasoning generated (~1 second)

This is the opposite extreme of the infinite thinking loop described in this issue:

Failure mode	Description
Infinite thinking loop (Issue #1)	Channel opens, model never exits, generation runs indefinitely
Empty thinking block (new)	Channel opens, model generates nothing, closes in ~1 second
Normal behavior	Channel opens, model reasons for N tokens, closes, produces response

Both failure modes appear to stem from the same root cause: the model's reasoning channel has no reliable entry/exit condition. In the infinite loop case it never exits; in the empty block case it exits immediately without engaging the reasoning pathway at all.

The <|channel>thought token appearing as literal text in the output suggests the model is not correctly transitioning into the reasoning generation state after opening the channel — it emits the channel marker and then immediately hits the exit condition, possibly because the thinking prompt context is malformed or incomplete on that particular inference call.

Environment:

• Model: google/gemma-4-12B-it Q8_0 GGUF
• Backend: llama.cpp via Jan 0.8.4 (Vulkan, Windows)
• Hardware: AMD Radeon RX 9060 XT (RDNA4)
• Also observed in LM Studio (same model, same hardware)

Related Jan issue tracking UI-side thinking inconsistency: janhq/jan#8530

GitHub: titansonico10

[soenning-ai]

Try a workaround in your pipe. I have the same problems and mostly get rid of it with retrys. Max 3 retrys. 2 retrys with exactly the same context. 3rd retry with an additional add "Retry because System run out of tokens." break the loop with a little shift in the attention.