Repository navigation
feat: let a served Tesseract enable several GPU transports at once - #925
dionhaefner wants to merge 10 commits into
Conversation
gpu_transport now takes one transport, a list, or a comma-separated string (as in TESSERACT_GPU_TRANSPORT). The server offers all of them, and a client that spawned it requests the first. cuda_ipc is still the only transport, but the places a new one slots in are now explicit.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #925 +/- ##
==========================================
+ Coverage 87.30% 88.21% +0.91%
==========================================
Files 57 57
Lines 7855 7902 +47
Branches 1213 1224 +11
==========================================
+ Hits 6858 6971 +113
+ Misses 714 708 -6
+ Partials 283 223 -60 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Benchmark ResultsBenchmarks use a no-op Tesseract to measure pure framework overhead. 🚀 0 faster, ✅ No significant performance changes detected. Full results
Ran 3 baseline and 3 current passes, alternating between the two. A benchmark is flagged only if every current pass differs from every baseline pass by more than 10% in the same direction. Times are the median over passes of each pass's median. Spread is the range of the per-pass medians on the noisier side, relative to its median.
|
…nsports # Conflicts: # tesseract_core/runtime/config.py # tesseract_core/runtime/file_interactions.py # tesseract_core/sdk/engine.py # tesseract_core/sdk/tesseract.py
There was a problem hiding this comment.
Okay, technically this seems fine. but to be honest I am not a big fan of this change.
Using cuda_ipc over nixl on single GPU shaves off a handful of ms at most. And the only real use case for this feature is a single tesseract being called both by a client on the same host as well as by a client on a different GPU machine in the same cluster, simultaneously. This sounds quite exotic.
To me this does not sound like a great trade off for additional complexity of the API. Help that no longer shows options, lists as inputs on CLI that i always find clumsy and confusing. Even handling of "none" I am not clear about - can it be a part of the list?
This might still be necessary to introduce, but I'd only do it if we had clear motivation to do so. At which point I hand over to you @dionhaefner - it this a proactive change or we already have a use case for it that would benefit?
These GPU features exist to shave off handfuls of ms in general. We have shipped several similar optimizations already (and I'd ship more if I could).
It is exotic and I have no idea when anyone would use this.
Agree we should fix
It's not necessary by any means. If anything this is a way to close a mismatch between internal and external APIs: we already support lists internally, and I see no good reason to restrict what users are allowed to do. If there are real UX regressions introduced here that can't be fixed easily I'm ok with closing this. |
|
okay! given my biggest grudges are with UX, can we please improve the CLI help and verify/streamline handling of "none"? then i have no arguments basically
i guess... i always find it unclear when CLI argument accepts a list. should i use comma separated string, or semicolon, or space? or specify same arg multiple times with different values? does the order matter? even if docs tell me what to do, i personally find it mentally harder to deal with. i might be a minority here |
…n runtime help 'none' is always accepted, so listing it next to other transports had no effect and silently ignored a likely intent. It is now only valid on its own. tesseract-runtime's --gpu-transport is repeatable like tesseract serve's and lists its choices in --help again. Comma-separated values still work, as from_source passes the setting through the environment. Invalid config values now raise a usage error instead of a pydantic traceback.
…GPU_TRANSPORT Click passes an empty tuple for an absent repeatable option when the global options are applied before the API import, which set gpu_transport to () and made a served Tesseract accept only 'none'.
|
@apaleyes should be ready for another round! |
| ), | ||
| (["--gpu-transport", "cuda_ipc,nixl"], ("cuda_ipc", "nixl")), | ||
| (["--gpu-transport", "none"], ()), | ||
| (["--gpu-transport", "none", "--gpu-transport", "nixl"], None), |
There was a problem hiding this comment.
why doesn't this test case fire "cannot be combined with other transports" error?
There was a problem hiding this comment.
it does fire, None is the error case. made that clearer by moving it to a separate test.
Clarify the gpu_transport docstring, reword the missing-device error, and move the rejected 'none' combination into its own test.
|
good to merge once CI is green, thanks! |
Relevant issue or PR
Follow-up to a review discussion on #723, which adds
nixlas a second GPU transport. A server should be able to offercuda_ipcto clients on its own host andnixlto clients elsewhere, without picking one.Description of changes
gpu_transportnow accepts one transport, a list, or a comma-separated string:The server enables and advertises every transport listed. The Tesseract that spawned it requests the first one on its calls, the same as it requests the single transport today. Other clients pick a transport per call as before, with
with_encodingorresolve_gpu_transport.Most of the stack already handled several transports:
x-supported-gpu-transportsis a list,negotiate_encodingpicks per request, andresolve_gpu_transportwalks a preference list. Only the config held a single value. It's now a tuple of enabled transports (empty meansnone).cuda_ipcis still the only transport, so for now this changes no behavior beyond accepting the list forms. The places a new transport slots in are now explicit:gpu_transport_typein the runtime,GPU_TRANSPORTSinsdk/serving.py, and a per-transport block inengine.servefor container setup (--ipc=hostforcuda_ipc).Two details reviewers might trip over:
tesseract-runtime --gpu-transportnow takes a string, as dict-valued config already does, so its--helpno longer lists the choices. Invalid names are still rejected when the config is validated.with_encoding(gpu_transport=...)still takes a single transport, since a call uses exactly one.Testing done
runtime_config, empty list), thefrom_sourcechild/client agreement, andnegotiate_encodingwith a comma-separated config.serve()uses to hand config to uvicorn.tests/runtime_tests,tests/sdk_tests,tests/test_cuda_ipc_cpu.py) pass locally on macOS.