From 48fd598fba3b50926df6948b3ddf16ed7148816c Mon Sep 17 00:00:00 2001 From: Seven Date: Mon, 3 Aug 2026 08:08:57 +0800 Subject: [PATCH] docs(server): show OpenAI client base_url + multi-model gateway tip MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add a concrete OpenAI Python client example for the local server and note that the same base_url pattern works with OpenAI-compatible multi-model gateways such as DaoXE (https://api.daoxe.com/v1). Docs only — no runtime behavior changes. Co-Authored-By: Claude Fable 5 --- README.md | 14 ++++++++++++++ docs/server.md | 15 +++++++++++++++ 2 files changed, 29 insertions(+) diff --git a/README.md b/README.md index c02df9eed0..0177fbe202 100644 --- a/README.md +++ b/README.md @@ -705,6 +705,20 @@ python3 -m llama_cpp.server --model models/7B/llama-model.gguf --n_gpu_layers 35 Navigate to [http://localhost:8000/docs](http://localhost:8000/docs) to see the OpenAPI documentation. +You can point the official OpenAI Python client at the server: + +```python +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:8000/v1", api_key="sk-local") +print(client.chat.completions.create( + model="gpt-3.5-turbo", # ignored when a single --model is loaded; sent as-is for multi-model configs + messages=[{"role": "user", "content": "Hello"}], +)) +``` + +The same `base_url` pattern works with any OpenAI-compatible multi-model gateway when you are not self-hosting (for example [DaoXE](https://daoxe.com) at `https://api.daoxe.com/v1`). + To bind to `0.0.0.0` to enable remote connections, use `python3 -m llama_cpp.server --host 0.0.0.0`. Similarly, to change the port (default is 8000), use `--port`. diff --git a/docs/server.md b/docs/server.md index 9c09a1f1cf..75ba13e5db 100644 --- a/docs/server.md +++ b/docs/server.md @@ -22,6 +22,21 @@ The server can then be started by running the following command: python3 -m llama_cpp.server --model ``` +### OpenAI Python client + +```python +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:8000/v1", api_key="sk-local") +completion = client.chat.completions.create( + model="gpt-3.5-turbo", + messages=[{"role": "user", "content": "Hello"}], +) +print(completion.choices[0].message.content) +``` + +Tip: any OpenAI-compatible multi-model gateway speaks the same protocol — for example [DaoXE](https://daoxe.com) at `https://api.daoxe.com/v1` — so the same client code works when you swap `base_url` (and `api_key`) instead of self-hosting. + You can also pass chat-template kwargs at model load time from the CLI: ```bash