Repository navigation
types: expose prompt_eval_cached_count on generate/chat responses - #781
Open
Uneadlight wants to merge 1 commit into
Open
Uneadlight wants to merge 1 commit into
Uneadlight wants to merge 1 commit into
Conversation
The server reports how many prompt tokens were served from the prompt cache, but BaseGenerateResponse had no field for it. Because SubscriptableBaseModel does not allow extra fields, pydantic silently dropped the value, so users could neither read it nor see it in model_dump(). Verified locally against ollama 0.40.13: POST /api/generate returns "prompt_eval_cached_count": 0 right after prompt_eval_count. Closes ollama#773
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix #773 — expose
prompt_eval_cached_counton generate/chat responsesProblem
GenerateResponse/ChatResponsesilently drop the server'sprompt_eval_cached_countfield, so callers cannot tell how many prompt tokens were served from the prompt cache.
Reported in #773 (and duplicated in #779).
Root cause
BaseGenerateResponse(inollama/_types.py) declares every numeric field explicitly andSubscriptableBaseModeldoes not setmodel_config = ConfigDict(extra='allow'), so pydantic v2uses its default
extra='ignore'and discards the unknown key on validation.Reproduction (local, ollama 0.40.13,
qwen2.5:3b)Two identical
generatecalls through the SDK after this change:Before the change both calls raised
AttributeError/ returnedNonefor that attribute,even though the raw HTTP response already carried
"prompt_eval_cached_count": 0.Change
ollama/_types.py— one field onBaseGenerateResponse(inherited by both response types):Optionalkeeps compatibility with older servers that do not send the field.Tests
Added to
tests/test_type_serialization.py:test_generate_response_keeps_prompt_eval_cached_count— attribute,[]access,in, andmodel_dump()test_chat_response_keeps_prompt_eval_cached_counttest_prompt_eval_cached_count_is_optional— a payload without the field still parses, andexclude_nonedoes not emit itResults:
tests/test_type_serialization.py→ 21 passed (2 pre-existing failures from thismachine's environment, see below); full suite → 112 passed, 8 failed (same 8 as before the change).
Why not
extra='allow'extra='allow'would accept any unknown key and would surface it throughmodel_dump(),which changes the public shape of every response object and could leak unrelated server fields.
Declaring the one documented field keeps the model explicit and type-checked.
Environment note (not a repository bug)
The 8 failures in this run are specific to this machine and unrelated to the change:
ValueError: Invalid image data, expected base64 string or path to image file— the localtemp directory cannot create files from a Python process (
tempfile.mkstemphangs), so thetest fixtures produce empty image files.
PermissionErrorintest_client_create_blob*— same temp-directory cause.They fail identically on a clean checkout with no local modifications.