You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: RELEASE_NOTES-v3.md
+27-9Lines changed: 27 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,10 +1,20 @@
1
1
# v3 Release Notes
2
2
3
+
## Rewritten for extensibility
4
+
5
+
A major rewrite of llms has been completed to make it extensible and allow for easy addition of new features and providers. Including built-in UI components which have been [refactored into modules](https://github.com/ServiceStack/llms/tree/main/llms/ui/modules) utilizing the same extensibility APIs any extension will be able to use. In addition to adding new features, extensions are also to replace existing components by registering new components with the same name.
6
+
7
+
Likewise, the server has adopted a server extension model, where major provider implementations can now be dropped into the [llms/providers](https://github.com/ServiceStack/llms/tree/main/llms/providers) folder where it will be automatically loaded and registered at runtime.
8
+
9
+
This allows [main.py](https://github.com/ServiceStack/llms/blob/main/llms/main.py) to continue to retain a lean functional core in a single file whislt still being extensible.
10
+
3
11
## New Provider Configuration Model
4
12
5
-
The internal provider configuration is now a **superset** of `models.dev/api.json`. Definitions are merged, allowing you to simply enable providers using `"enabled": true` while inheriting all standard provider configurations from `models.dev`.
13
+
The most disruptive change is the migration to utilize the same [models.dev](https://models.dev) open provider and model catalogue as used and maintained by [OpenCode](https://opencode.ai).
14
+
15
+
llms provider configuration is now a **superset** of `models.dev/api.json` where its definitions are merged, allowing you to enable providers using just `"enabled": true` where it inherits all standard provider configurations from `models.dev`.
6
16
7
-
The switch to models.dev greatly expands our model selection to not more than 530 providers from 23 different providers. Including new support for:
17
+
The switch to models.dev greatly expands our model selection to more than 530 models from 23 different providers. Including new support for:
8
18
9
19
- Alibaba (39 models)
10
20
- Chutes (56 models)
@@ -24,11 +34,11 @@ Please raise an issue to add support for any missing providers from [models.dev]
24
34
25
35
### Up to Date Providers
26
36
27
-
If your providers.json is older than 1 day, it will be automatically updated when you run `llms`. You can also use the `--update-providers` command at anytime to update your local `providers.json` with the latest provider list from `models.dev`.
37
+
This also allows `llms` to automatically update your `providers.json` with the latest provider list from `models.dev` daily. You can also use the `--update-providers` command at anytime to update your local `providers.json` with the latest provider list from `models.dev`.
28
38
29
-
It filters and saves only the providers that are referenced in your `llms.json`. Any additional providers you want to use that are not included in `models.dev` can be added to your `~/.llms/providers-extra.json` which will be merged into your `providers.json` when updated.
39
+
`llms` filters and saves only the providers that are referenced in your `llms.json`. Any additional providers you want to use that are not included in `models.dev` can be added to your `~/.llms/providers-extra.json` which will be merged into your `providers.json` when updated.
30
40
31
-
This optimization keeps your local configuration file lightweight by only containing the providers you actually use.
41
+
This optimization keeps your local configuration file lightweight by only containing the providers that are available for use.
32
42
33
43
## New Model Selector UI
34
44
@@ -128,7 +138,9 @@ Use `map_models` to explicitly whitelist and map specific models. Only the model
128
138
```
129
139
130
140
### 4. Custom Provider & Model Definitions
131
-
You can still fully define custom providers and models that aren't in `models.dev`. For example, adding Mistral's free `codestral` endpoint:
141
+
142
+
You can still fully define custom OpenAI compatible providers and models that aren't in `models.dev`. For example, adding Mistral's free `codestral` endpoint:
143
+
132
144
```json
133
145
"codestral": {
134
146
"enabled": true,
@@ -147,6 +159,8 @@ You can still fully define custom providers and models that aren't in `models.de
147
159
}
148
160
```
149
161
162
+
If you just want to enable access to new models for existing providers whilst you're waiting for them to be added to models.dev, you can add them to your ``~/.llms/providers-extra.json` where they'll be merged into your `providers.json` when updated.
163
+
150
164
### 5. NPM SDK Alignment
151
165
The provider configuration is now closely aligned with the `models.dev` npm configuration. The `"npm"` field is used to map the provider configuration to the correct Python provider implementation. This generic mapping allows for flexible provider support, including **Anthropic Chat Completion** requests, which are correctly handled for both **Anthropic** and **MiniMax** providers.
152
166
@@ -165,16 +179,20 @@ Extensions can be installed from GitHub or by creating a local folder:
165
179
Extensions are Python modules that plug into the server lifecycle using special hooks defined in their `__init__.py`:
-**`__install__(ctx)`**: Enhance the server instance (e.g., add routes, register providers, filters, etc). `ctx` gives you access to the `ExtensionContext`.
182
+
-**`__install__(ctx)`**: Enhance the server instance (e.g., add routes, register providers, request/response filters, etc). `ctx` gives you access to the `ExtensionContext`.
169
183
-**`__run__(ctx)`**: Execute custom logic when running in CLI mode.
170
184
171
185
### How it Works (UI)
172
-
Extensions can also include a frontend component.
186
+
Extensions can also just include a frontend component.
173
187
1.**Placement**: Add a `ui` folder within your extension directory.
174
188
2.**Access**: Files in this folder are automatically served at `/ext/<extension_name>/*`.
175
189
3.**Integration**: Create a `ui/index.mjs` file. This is the entry point and must export an `install` function:
176
190
177
191
```javascript
192
+
constMyComponent= {
193
+
template:`...`
194
+
}
195
+
178
196
// ui/index.mjs
179
197
exportdefault {
180
198
install(ctx) {
@@ -315,7 +333,7 @@ v3 includes built-in support for audio generation on Google's new TTS models:
315
333
-**Gemini 2.5 Flash Preview TTS**
316
334
-**Gemini 2.5 Pro Preview TTS**
317
335
318
-
This is available both in the UI and on the command-line using `--out audio`, e.g:
336
+
This is available in both the UI and on the command-line using `--out audio`, e.g:
0 commit comments