Error occurs when koboldCpp acts as a server and connects to OpenCode via OpenAI-compatible API

[GarrisonDivision]

Description

When using koboldCpp as a server to connect to OpenCode through an OpenAI-compatible API, errors appear during usage. The system lags, performance is not fully utilized, and generation speed is reduced compared to using KoboldAI Lite. The function remains usable but significantly affects overall efficiency.

Steps to Reproduce

1. Start koboldCpp as an API server
2. Configure the OpenAI-compatible API endpoint in OpenCode
3. Send code generation or invocation requests
4. Errors appear with noticeable lag and slow generation speed

Error Message

q8日志.txt

 

Impact

AI model calls from OpenCode via koboldCpp experience lag and slow response, reducing work efficiency.

Temporary Solution

No effective temporary solution. Performance and API compatibility need to be investigated and optimized.

Notes

This issue is reproducible consistently. It does not break functionality but severely impacts generation speed and user experience.Describe the Issue**
A clear and detailed description of what the issue is, and how to duplicate it (if applicable).

Additional Information:
Please provide as much relevant information about your setup as possible, such as the Operating System, CPU, GPU, KoboldCpp Version, and relevant logs (helpful to include the launch params from the terminal output, flags and crash logs)

[wbruna]

Please provide as much relevant information about your setup as possible, such as the Operating System, CPU, GPU, KoboldCpp Version, and relevant logs (helpful to include the launch params from the terminal output, flags and crash logs)

This part is kind of important. Without knowing which model you are running, with which parameters, and what are your system's specs, it's impossible to know if "system lags, performance is not fully utilized, and generation speed is reduced" aren't to be expected (for instance, if your context is overflowing available VRAM). The logs will also show if e.g. OpenCode isn't simply overwhelming your server with requests.

The provided message:

q8.txt

looks like a request+answer, or just the answer. A rough translation by a local model:

Details

Solve the equation:

3(x − 4) + 5 = 2(x + 1) − 3

Step 1: Expand the brackets

3x − 12 + 5 = 2x + 2 − 3

Step 2: Combine like terms (simplify both sides)
Left side: −12 + 5 = −7, resulting in 3x − 7
Right side: 2 − 3 = −1, resulting in 2x − 1

The equation becomes:

3x − 7 = 2x − 1

Step 3: Move terms
Move the terms containing x to the left, and constant terms to the right (or subtract 2x from both sides):

3x − 2x − 7 = −1
x − 7 = −1

Or written as:

3x − 2x = −1 + 7

Step 4: Solve for x

x = −1 + 7
x = 6

Verification:
When x = 6:
Left side: 3(6 − 4) + 5 = 3 × 2 + 5 = 6 + 5 = 11
Right side: 2(6 + 1) − 3 = 2 × 7 − 3 = 14 − 3 = 11
Both sides are equal; the solution is correct.

Answer: x = 6

It would be more useful to us if we could see this together with other request details, on KoboldCpp logs. Could you provide the complete logs of this run (say, redirecting the output to a text file)?

[GarrisonDivision]

I have re-uploaded it.

[LostRuins]

Hi @GarrisonDivision , looking at your logs, you are using a Tesla P4 which has 8GB of VRAM. Therefore you can do a full offload, but you are currently only partially offloading 17 layers, because you are trying to load 131072 context. Also, you are using a q4 KV type (--quantkv 2) which tends to drastically lower generation speed.

All that context space is wasted since you only used 5326/131072

My suggestion:

• Use f16 KV type instead
• Load only 32k contextsize

It will be a lot faster. Please try this.

[LostRuins]

Please reopen if the issue persists