diff --git a/README.md b/README.md index c02df9eed0..0177fbe202 100644 --- a/README.md +++ b/README.md @@ -705,6 +705,20 @@ python3 -m llama_cpp.server --model models/7B/llama-model.gguf --n_gpu_layers 35 Navigate to [http://localhost:8000/docs](http://localhost:8000/docs) to see the OpenAPI documentation. +You can point the official OpenAI Python client at the server: + +```python +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:8000/v1", api_key="sk-local") +print(client.chat.completions.create( + model="gpt-3.5-turbo", # ignored when a single --model is loaded; sent as-is for multi-model configs + messages=[{"role": "user", "content": "Hello"}], +)) +``` + +The same `base_url` pattern works with any OpenAI-compatible multi-model gateway when you are not self-hosting (for example [DaoXE](https://daoxe.com) at `https://api.daoxe.com/v1`). + To bind to `0.0.0.0` to enable remote connections, use `python3 -m llama_cpp.server --host 0.0.0.0`. Similarly, to change the port (default is 8000), use `--port`. diff --git a/docs/server.md b/docs/server.md index 9c09a1f1cf..75ba13e5db 100644 --- a/docs/server.md +++ b/docs/server.md @@ -22,6 +22,21 @@ The server can then be started by running the following command: python3 -m llama_cpp.server --model ``` +### OpenAI Python client + +```python +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:8000/v1", api_key="sk-local") +completion = client.chat.completions.create( + model="gpt-3.5-turbo", + messages=[{"role": "user", "content": "Hello"}], +) +print(completion.choices[0].message.content) +``` + +Tip: any OpenAI-compatible multi-model gateway speaks the same protocol — for example [DaoXE](https://daoxe.com) at `https://api.daoxe.com/v1` — so the same client code works when you swap `base_url` (and `api_key`) instead of self-hosting. + You can also pass chat-template kwargs at model load time from the CLI: ```bash