Testing issues

[fidecastro]

@fidecastro thanks for your efforts! I am testing and ran into some issues.

1. I setup the vision connector server and when I send prompts through the api, using open-webui in this example, the prompt is sent to the llama binary without quotations and so they are not parsed correctly (I do not get a reply)-

The command works when I manually exit the server and run it directly.

2. Due to the Widespread errors with Nvidia's latest GPU drivers 572.*, I have reverted back to older drivers from December and they are on Cuda 12.7 so below the current spec of the Docker stage1 image. I believe it's a safe bet to just revise to use cuda 12.4 (that's what Llama.cpp recommends based on their compiled binaries).

In Dockerfile:
Stage 1: Build environment
ARG BASE_CUDA_DEV_CONTAINER=nvidia/cuda:12.4.1-cudnn-devel-ubuntu22.04
ARG BASE_CUDA_RUN_CONTAINER=nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04

3. I am in a dual GPU setup with a 3090 + 4090. The build script did not detect both gpus and went to cuda architecture 89 as the hard-coded default and then the image won't run. I went back and changed it in the Dockerfile and build-docker-container.sh to "89;86" and things were fine.

Could something be done to adjust the cuda architecture detection in a multi-gpu setup to mitigate this issue?

Originally posted by @ColumbusAI in #1

[fidecastro]

@ColumbusAI -- can you show me the error you're getting? How are you trying to connect to it?

The command without the quotes after -p, weirdly enough, should not be creating any issues. This is because I'm running it as a subprocess, the quotes should only be necessary if the CLI was being called directly on a shell terminal. The print you are seeing is a reconstitution of the subprocess being sent, which seems to be correctly assembled.

Try launching python llama_vision_connector.py server and then running python examples/test_vision_connector.py on another terminal. What is the output you get?

Wow, I didn't know about the driver snafu. You can 100% roll back to an older CUDA; I believe 12.4 is good enough. Just bear in mind I used ubuntu24.04, so there may be some dependency issue flaring up are greater if you go with 22.04. If you do use ubuntu22.04, let me know how it goes and maybe we add your report to the README somewhere.

[ColumbusAI]

It's still not responding to me in either OAI server endpoint or the test example.

Here is the log from the terminal-

terminal_output.log

I did as you asked and opened the same container in another terminal here is the output:

docker exec -it 90317c1d928a /bin/sh
# python examples/test_vision_connector.py

--- Running Direct Call Test ---

=== Testing Direct Vision Model Interaction (using get_response) ===
Initializing LlamaVisionConnector...
No model key specified. Defaulting to first model: 'QWEN2_5_VL_72B'
auto_start is False. Server process not started automatically.

Testing with image: /workspace/examples/test_images/sample.jpg
Prompt: Describe this image in detail.
Running command: llama-qwen2vl-cli -m /workspace/models/r/Qwen2.5-VL-72B-Instruct-IQ4_NL.gguf -fa -sm row --mmproj /workspace/models/r/qwen2.5-vl-72b-instruct-vision.gguf --temp 0.1 -ngl 99 --image /workspace/examples/test_images/sample.jpg -p Describe this image in detail.

[ColumbusAI]

OK the issue was my fault -- I was using qwen2.5 vl GGUF and that's not supported yet by llama.cpp.

I am using qwen2vl gguf and it worked but I'm noticing that I can't have conversation about the image. Is this the expected behavior?

Particularly, I'm noticing that the LLM is repeating itself (with slight variances) and not replying correctly to the prompt.

[Server STDOUT]: {
[Server STDOUT]: "role": "assistant",
[Server STDOUT]: "content": "This is a picture of a rabbit. The rabbit has a gray fur coat and is sitting on a white background."
[Server STDOUT]: },
[Server STDOUT]: {
[Server STDOUT]: "role": "user",
[Server STDOUT]: "content": "which way is the rabbit facing?"
[Server STDOUT]: },
[Server STDOUT]: {
[Server STDOUT]: "role": "assistant",
[Server STDOUT]: "content": "This is a picture of a rabbit. The rabbit has a gray fur coat and is sitting with its ears upright."
[Server STDOUT]: },
[Server STDOUT]: {
[Server STDOUT]: "role": "user",
[Server STDOUT]: "content": "what else can you tell me?"
[Server STDOUT]: }

[fidecastro]

I see what's happening. This is an issue with the llama CLI itself -- it does not allow more than one inquiry with the image. The way around this must be for LlamaVisionConnector to send all messages back to the CLI instead of just the prompt. I'll work on that.

[fidecastro]

(Important side note: the CLI also has a limitation of only receiving one single image, so it can only be used with one image.)