Skip to content

FIX: Calculate accurate UTF-8 byte length for content-length in HTTPTarget - #3008

Open
Vardhman Gupta (Kaap10) wants to merge 1 commit into
microsoft:mainfrom
Kaap10:fix/http-target-content-length-byte-count
Open

Vardhman Gupta (Kaap10) wants to merge 1 commit into
microsoft:mainfrom
Kaap10:fix/http-target-content-length-byte-count

Conversation

@Kaap10

Copy link
Copy Markdown
Contributor

Description

While testing HTTPTarget with non-ASCII and multilingual prompts (such as Chinese characters and emojis), I noticed that requests were getting truncated on the receiving server.

Looking into HTTPTarget._send_prompt_to_target_async, the content-length header was being calculated using len(http_body). In Python, len() on a string returns the character count instead of the actual encoded byte count. Per RFC 9110, Content-Length must be the number of octets/bytes. For multi-byte characters (e.g. "你好" which is 2 characters but 6 UTF-8 bytes), sending content-length: 2 caused the payload to be cut short.

Summary of Changes:

  • Updated _send_prompt_to_target_async to use len(http_body.encode("utf-8")) for string bodies so the byte count is accurate.
  • Added handling for bytes/bytearray bodies.
  • Avoided adding content-length when http_body is empty (e.g., bodyless GET requests).
  • Used lowercase "content-length" to keep dictionary keys consistent with parse_raw_http_request.

Tests and Documentation

  • Added test_send_prompt_async_content_length_utf8_bytes to verify multi-byte characters calculate the correct byte length.
  • Added test_send_prompt_async_content_length_recalculated to check header updates when overriding raw requests.
  • Added test_send_prompt_async_content_length_raw_bytes for raw byte payload testing.
  • Updated existing test_http_target.py mock assertions.
  • Verified with pytest tests/unit/prompt_target/ (all passed) and ruff check.

@Kaap10

Copy link
Copy Markdown
Contributor Author

This is ready for review. Thanks!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant