Skip to content

feat: let a served Tesseract enable several GPU transports at once - #925

Open
dionhaefner wants to merge 10 commits into
mainfrom
feat/multiple-gpu-transports
Open

dionhaefner wants to merge 10 commits into
mainfrom
feat/multiple-gpu-transports

Conversation

@dionhaefner

Copy link
Copy Markdown
Contributor

Relevant issue or PR

Follow-up to a review discussion on #723, which adds nixl as a second GPU transport. A server should be able to offer cuda_ipc to clients on its own host and nixl to clients elsewhere, without picking one.

Description of changes

gpu_transport now accepts one transport, a list, or a comma-separated string:

tess = Tesseract.from_image("my-gpu-tesseract", gpus=["all"], gpu_transport=["cuda_ipc"])
$ tesseract serve my-gpu-tesseract --gpus all -e TESSERACT_GPU_TRANSPORT=cuda_ipc

The server enables and advertises every transport listed. The Tesseract that spawned it requests the first one on its calls, the same as it requests the single transport today. Other clients pick a transport per call as before, with with_encoding or resolve_gpu_transport.

Most of the stack already handled several transports: x-supported-gpu-transports is a list, negotiate_encoding picks per request, and resolve_gpu_transport walks a preference list. Only the config held a single value. It's now a tuple of enabled transports (empty means none).

cuda_ipc is still the only transport, so for now this changes no behavior beyond accepting the list forms. The places a new transport slots in are now explicit: gpu_transport_type in the runtime, GPU_TRANSPORTS in sdk/serving.py, and a per-transport block in engine.serve for container setup (--ipc=host for cuda_ipc).

Two details reviewers might trip over:

  • tesseract-runtime --gpu-transport now takes a string, as dict-valued config already does, so its --help no longer lists the choices. Invalid names are still rejected when the config is validated.
  • with_encoding(gpu_transport=...) still takes a single transport, since a call uses exactly one.

Testing done

  • Extended the existing parametrized tests: engine precedence (list kwarg, list in runtime_config, empty list), the from_source child/client agreement, and negotiate_encoding with a comma-separated config.
  • Added a test that the list survives the trip through the environment that serve() uses to hand config to uvicorn.
  • Fast suites (tests/runtime_tests, tests/sdk_tests, tests/test_cuda_ipc_cpu.py) pass locally on macOS.
  • Not run locally: the GPU end-to-end tests, which need CUDA. They only use the single-string form, which keeps its behavior, so CI's GPU jobs should cover them.

gpu_transport now takes one transport, a list, or a comma-separated string
(as in TESSERACT_GPU_TRANSPORT). The server offers all of them, and a client
that spawned it requests the first. cuda_ipc is still the only transport, but
the places a new one slots in are now explicit.
@codecov

codecov Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 86.58537% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.21%. Comparing base (458a24a) to head (286b1d4).

Files with missing lines Patch % Lines
tesseract_core/runtime/cli.py 83.33% 2 Missing and 1 partial ⚠️
tesseract_core/sdk/tesseract.py 84.21% 1 Missing and 2 partials ⚠️
tesseract_core/runtime/config.py 84.61% 1 Missing and 1 partial ⚠️
tesseract_core/sdk/serving.py 88.23% 1 Missing and 1 partial ⚠️
tesseract_core/sdk/engine.py 87.50% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #925      +/-   ##
==========================================
+ Coverage   87.30%   88.21%   +0.91%     
==========================================
  Files          57       57              
  Lines        7855     7902      +47     
  Branches     1213     1224      +11     
==========================================
+ Hits         6858     6971     +113     
+ Misses        714      708       -6     
+ Partials      283      223      -60     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@PasteurBot

PasteurBot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Benchmark Results

Benchmarks use a no-op Tesseract to measure pure framework overhead.

🚀 0 faster, ⚠️ 0 slower, ✅ 18 unchanged

✅ No significant performance changes detected.

Full results
Benchmark Baseline Current Change Spread Status
api/apply_0 0.434ms 0.439ms +1.2% 3.6% ✅
api/apply_1,000,000 0.435ms 0.437ms +0.5% 2.7% ✅
http/apply_0 3.009ms 3.023ms +0.4% 4.0% ✅
http/apply_1,000,000 97.589ms 95.050ms -2.6% 3.3% ✅
http-shmem/apply_0 4.212ms 4.143ms -1.7% 2.4% ✅
http-shmem/apply_1,000,000 10.668ms 10.426ms -2.3% 11.1% ✅
roundtrip/base64_0 0.068ms 0.067ms -0.4% 2.7% ✅
roundtrip/base64_1,000,000 8.723ms 10.266ms +17.7% 21.7% ✅
roundtrip/base64+lz4_0 0.072ms 0.072ms +0.1% 3.3% ✅
roundtrip/base64+lz4_1,000,000 35.013ms 35.152ms +0.4% 4.6% ✅
roundtrip/binref_0 0.403ms 0.408ms +1.4% 3.7% ✅
roundtrip/binref_1,000,000 3.227ms 2.854ms -11.5% 24.4% ✅
roundtrip/json_0 0.063ms 0.062ms -0.4% 2.7% ✅
roundtrip/json_1,000,000 223.406ms 224.100ms +0.3% 1.4% ✅
subprocess/apply_0 1.767ms 1.891ms +7.0% 9.2% ✅
subprocess/apply_1,000,000 85.457ms 85.069ms -0.5% 9.9% ✅
subprocess-shmem/apply_0 2.485ms 2.420ms -2.6% 2.9% ✅
subprocess-shmem/apply_1,000,000 7.852ms 8.122ms +3.4% 11.0% ✅

Ran 3 baseline and 3 current passes, alternating between the two. A benchmark is flagged only if every current pass differs from every baseline pass by more than 10% in the same direction. Times are the median over passes of each pass's median. Spread is the range of the per-pass medians on the noisier side, relative to its median.

…nsports

# Conflicts:
#	tesseract_core/runtime/config.py
#	tesseract_core/runtime/file_interactions.py
#	tesseract_core/sdk/engine.py
#	tesseract_core/sdk/tesseract.py
@dionhaefner dionhaefner added the next-release Issues/PRs that should be addressed before the next release is made. label Nov 12, 2026

@apaleyes apaleyes left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, technically this seems fine. but to be honest I am not a big fan of this change.

Using cuda_ipc over nixl on single GPU shaves off a handful of ms at most. And the only real use case for this feature is a single tesseract being called both by a client on the same host as well as by a client on a different GPU machine in the same cluster, simultaneously. This sounds quite exotic.

To me this does not sound like a great trade off for additional complexity of the API. Help that no longer shows options, lists as inputs on CLI that i always find clumsy and confusing. Even handling of "none" I am not clear about - can it be a part of the list?

This might still be necessary to introduce, but I'd only do it if we had clear motivation to do so. At which point I hand over to you @dionhaefner - it this a proactive change or we already have a use case for it that would benefit?

Comment thread tesseract_core/runtime/config.py Resolved
@dionhaefner

dionhaefner commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Using cuda_ipc over nixl on single GPU shaves off a handful of ms at most.

These GPU features exist to shave off handfuls of ms in general. We have shipped several similar optimizations already (and I'd ship more if I could).

And the only real use case for this feature is a single tesseract being called both by a client on the same host as well as by a client on a different GPU machine in the same cluster, simultaneously. This sounds quite exotic.

It is exotic and I have no idea when anyone would use this.

To me this does not sound like a great trade off for additional complexity of the API. Help that no longer shows options, lists as inputs on CLI that i always find clumsy and confusing. Even handling of "none" I am not less clear about - can it be a part of the list?

Agree we should fix "none" handling. Passing flags multiple times to indicate a list is standard CLI practice, no? "Help that no longer shows options" should be fixable, too.

This might still be necessary to introduce, but I'd only do it if we had clear motivation to do so. At which point I hand over to you @dionhaefner - it this a proactive change or we already have a use case for it that would benefit?

It's not necessary by any means. If anything this is a way to close a mismatch between internal and external APIs: we already support lists internally, and I see no good reason to restrict what users are allowed to do.

If there are real UX regressions introduced here that can't be fixed easily I'm ok with closing this.

@apaleyes

apaleyes commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

okay! given my biggest grudges are with UX, can we please improve the CLI help and verify/streamline handling of "none"? then i have no arguments basically

Passing flags multiple times to indicate a list is standard CLI practice, no?

i guess... i always find it unclear when CLI argument accepts a list. should i use comma separated string, or semicolon, or space? or specify same arg multiple times with different values? does the order matter? even if docs tell me what to do, i personally find it mentally harder to deal with. i might be a minority here

…n runtime help

'none' is always accepted, so listing it next to other transports had no
effect and silently ignored a likely intent. It is now only valid on its own.

tesseract-runtime's --gpu-transport is repeatable like tesseract serve's and
lists its choices in --help again. Comma-separated values still work, as
from_source passes the setting through the environment. Invalid config
values now raise a usage error instead of a pydantic traceback.
…GPU_TRANSPORT

Click passes an empty tuple for an absent repeatable option when the global
options are applied before the API import, which set gpu_transport to () and
made a served Tesseract accept only 'none'.
@dionhaefner

Copy link
Copy Markdown
Contributor Author

@apaleyes should be ready for another round!

Comment thread docs/content/how-to/gpu-transport.md Resolved
Comment thread docs/content/how-to/gpu-transport.md Resolved
Comment thread tesseract_core/runtime/cuda/api.py Resolved
Comment thread tesseract_core/runtime/config.py Resolved
Comment thread tests/runtime_tests/test_runtime_cli.py Outdated
),
(["--gpu-transport", "cuda_ipc,nixl"], ("cuda_ipc", "nixl")),
(["--gpu-transport", "none"], ()),
(["--gpu-transport", "none", "--gpu-transport", "nixl"], None),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why doesn't this test case fire "cannot be combined with other transports" error?

@dionhaefner dionhaefner Nov 17, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it does fire, None is the error case. made that clearer by moving it to a separate test.

Clarify the gpu_transport docstring, reword the missing-device error,
and move the rejected 'none' combination into its own test.
@apaleyes

Copy link
Copy Markdown
Contributor

good to merge once CI is green, thanks!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

next-release Issues/PRs that should be addressed before the next release is made.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants