perf: use httptools for HTTP parsing - #2869
Open
ananthsub wants to merge 2 commits into
Open
Conversation
Gym depends on plain uvicorn rather than uvicorn[standard], so httptools was never installed and every server silently fell back to uvicorn's pure-Python h11 protocol implementation for HTTP parsing. Add httptools (C bindings to the llhttp parser) as a dependency and name it explicitly in the uvicorn config: uvicorn's "auto" selection only picks httptools when the wheel is importable and falls back to h11 silently, so an explicit http="httptools" makes a missing wheel fail loudly at startup instead of silently degrading. Measured on a SimpleResponsesAPIModel serving training-shaped bodies through the production middleware stack: +1-4% alone (response serialization dominates the server today), rising to +5-11% at high request rates once the orjson dispatch-serialization change lands. Wheels cover manylinux/musllinux aarch64 and x86_64 for cp310-cp314. No behavior change for well-formed traffic; llhttp is stricter than h11 on malformed requests. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
Use short sentences to explain why Gym explicitly selects httptools and how startup behaves when the dependency is unavailable. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
2 tasks
ananthsub
changed the title
bxyu-nvidia
approved these changes
Aug 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Gym depends on plain
uvicorn, sohttptoolsis not guaranteed to be installed and Uvicorn can silently fall back to its pure-Python h11 parser.This change adds
httptoolsas a dependency and explicitly selects it in the Uvicorn configuration. Explicit selection also turns a missing or incompatible wheel into a startup error instead of silently changing the server's parser.Performance methodology
For testing, added a synthetic benchmark that used a deterministic
SimpleResponsesAPIModelwhich returns cached, training-shaped dictionaries. This isolates Gym's dispatch, middleware, serialization, and HTTP serving costs from model generation.This was tested with #2867
The matrix covered:
The httptools-only variant improved median throughput by up to 4.2%. The result was neutral at 8,192 concurrency, where server saturation and scheduling variance dominated parser cost.
Test plan
uv run pytest tests/unit_tests/test_server_utils.py -quv run pre-commit run --files nemo_gym/server_utils.py pyproject.toml uv.lock