Repository navigation
Conversation
LiteLLM normalizes every provider's usage into prompt_tokens_details and completion_tokens_details, but the integration only read prompt, completion and total tokens. Read cached, cache-write and reasoning token counts from those details, falling back to LiteLLM's private cache fields. Fixes getsentry#5455
b47d6e7 to
10dbd45
Compare
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 10dbd45. Configure here.
| written = _int_or_none(getattr(details, attr, None)) | ||
| if written is not None: | ||
| return written | ||
| return _int_or_none(getattr(usage, "_cache_creation_input_tokens", None)) |
There was a problem hiding this comment.
Cache fallback records default zeros
Medium Severity
The fallbacks to _cache_read_input_tokens and _cache_creation_input_tokens treat LiteLLM's default 0 as real usage. Those private fields are initialized to 0 on every Usage object, so _int_or_none returns 0 and the span gets gen_ai.usage.input_tokens.cached and gen_ai.usage.input_tokens.cache_write even when the provider sent no cache data. The no-details test never constructs LiteLLM Usage, so this path is untested.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 10dbd45. Configure here.
chrikrah
left a comment
chrikrah
left a comment
There was a problem hiding this comment.
@4ktLuffy From my perspective this can be merged once the zeros go. blocking: Bugbot's cache-zeros thread reproduces with a real Usage:
# probe: _success_callback with response.usage = litellm.types.utils.Usage(prompt_tokens=10, completion_tokens=1, total_tokens=11)
$ pytest tests/integrations/litellm/test_zz_probe.py -q -s --no-cov # 10dbd45, litellm 1.106.4, Python 3.12
PROBE {'gen_ai.usage.input_tokens': 10, 'gen_ai.usage.input_tokens.cached': 0, 'gen_ai.usage.input_tokens.cache_write': 0, 'gen_ai.usage.output_tokens': 1, 'gen_ai.usage.total_tokens': 11}
# the same Usage built with a cache read of 2048 and a cache creation of 512:
# its details object holds cached 2048, cache write 512, cache creation 512
# PR body: "details missing ... no cached/cache-write/reasoning attributes"
# test_token_usage_without_details asserts these keys are absent, but builds its usage with MockCompletionResponse
# openai.py at 10dbd45: "Do not set token data if it is 0", input_tokens_cached = input_tokens_cached or None
$ pytest tests/integrations/litellm/test_litellm.py -q --no-cov
170 passed, 194 warnings, 8 errors in 57.79s # 10dbd45
4 failed, 166 passed, 194 warnings, 8 errors in 56.91s # litellm.py from ab069a5, tests kept
# not run: the 8 errors are test_multiple_providers fixtures (google-genai not installed), same on both
Could all three attributes skip 0, as openai.py does?
Description
LiteLLM normalizes every provider's usage into
prompt_tokens_detailsandcompletion_tokens_details(Anthropic'scache_read_input_tokens/cache_creation_input_tokensincluded), and the success callback receives thatUsageobject. The integration only read prompt, completion and total tokens, so cached, cache-write and reasoning tokens were dropped.This reads them from those details (falling back to LiteLLM's private
_cache_*fields) and sets them on the span next to the existing token fields.Tested:
tests/integrations/litellm/test_litellm.py:test_cached_and_reasoning_token_usage(sync)test_async_cached_and_reasoning_token_usage(acompletion)test_streaming_cached_and_reasoning_token_usage(stream=True, usage in the final chunk viastream_options={"include_usage": True})test_cache_write_token_usage_from_anthropic_shaped_usage(cache read + creation counts)test_token_usage_without_details(details missing,None, or holdingNone: no cached/cache-write/reasoning attributes)KeyError: 'gen_ai.usage.input_tokens.cached'and pass with this change; the last passes on both.py3.12-litellm-v1.77.7and onpy3.13-litellm-v1.96.0.ruff check,ruff format --checkandmypy sentry_sdkare clean.tox.ini(1.77.7 to latest, 1.104.0) plus the current pre-release (1.105.0rc1): cached, cache-write and reasoning tokens match the provider's numbers on all of them.To reproduce:
Issues