Skip to content

types: expose prompt_eval_cached_count on generate/chat responses - #781

Open
Uneadlight wants to merge 1 commit into
ollama:mainfrom
Uneadlight:fix/773-prompt-eval-cached-count
Open

Uneadlight wants to merge 1 commit into
ollama:mainfrom
Uneadlight:fix/773-prompt-eval-cached-count

Conversation

@Uneadlight

Copy link
Copy Markdown

Fix #773 — expose prompt_eval_cached_count on generate/chat responses

Problem

GenerateResponse / ChatResponse silently drop the server's prompt_eval_cached_count
field, so callers cannot tell how many prompt tokens were served from the prompt cache.

Reported in #773 (and duplicated in #779).

Root cause

BaseGenerateResponse (in ollama/_types.py) declares every numeric field explicitly and
SubscriptableBaseModel does not set model_config = ConfigDict(extra='allow'), so pydantic v2
uses its default extra='ignore' and discards the unknown key on validation.

Reproduction (local, ollama 0.40.13, qwen2.5:3b)

Two identical generate calls through the SDK after this change:

--- call 1 (cold) ---
prompt_eval_count       = 36
prompt_eval_cached_count= 0
--- call 2 (same prompt) ---
prompt_eval_count       = 36
prompt_eval_cached_count= 35

Before the change both calls raised AttributeError / returned None for that attribute,
even though the raw HTTP response already carried "prompt_eval_cached_count": 0.

Change

ollama/_types.py — one field on BaseGenerateResponse (inherited by both response types):

prompt_eval_cached_count: Optional[int] = None
'Number of prompt tokens that were served from the prompt cache instead of being evaluated.'

Optional keeps compatibility with older servers that do not send the field.

Tests

Added to tests/test_type_serialization.py:

  • test_generate_response_keeps_prompt_eval_cached_count — attribute, [] access, in, and model_dump()
  • test_chat_response_keeps_prompt_eval_cached_count
  • test_prompt_eval_cached_count_is_optional — a payload without the field still parses, and
    exclude_none does not emit it

Results: tests/test_type_serialization.py → 21 passed (2 pre-existing failures from this
machine's environment, see below); full suite → 112 passed, 8 failed (same 8 as before the change).

Why not extra='allow'

extra='allow' would accept any unknown key and would surface it through model_dump(),
which changes the public shape of every response object and could leak unrelated server fields.
Declaring the one documented field keeps the model explicit and type-checked.

Environment note (not a repository bug)

The 8 failures in this run are specific to this machine and unrelated to the change:

  • 6 × ValueError: Invalid image data, expected base64 string or path to image file — the local
    temp directory cannot create files from a Python process (tempfile.mkstemp hangs), so the
    test fixtures produce empty image files.
  • 2 × PermissionError in test_client_create_blob* — same temp-directory cause.

They fail identically on a clean checkout with no local modifications.

The server reports how many prompt tokens were served from the prompt
cache, but BaseGenerateResponse had no field for it. Because
SubscriptableBaseModel does not allow extra fields, pydantic silently
dropped the value, so users could neither read it nor see it in
model_dump().

Verified locally against ollama 0.40.13: POST /api/generate returns
"prompt_eval_cached_count": 0 right after prompt_eval_count.

Closes ollama#773

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GenerateResponse/ChatResponse silently drop the server's prompt_eval_cached_count field

1 participant