Summary
When ModelExpress is shared by workers in different RDMA network domains, the same model configuration can produce the same mx_source_id for source instances that are not mutually reachable over RDMA. A target worker can select a READY source from another domain because source selection currently matches model identity, worker rank, and accelerator compatibility, but does not
consider RDMA topology.
When the selected source is unreachable, the target falls back after the failure has already mutated the target model. The retry path reinitializes a second model while the original model is still held by the outer loader call.
The result is a near-double GPU allocation and the worker is killed by the OOM killer.
Environment
- Kubernetes deployment with shared ModelExpress metadata
- Multiple RDMA fabrics/domains in the same Kubernetes cluster
- SGLang ModelExpress loader
- NIXL with UCX backend
- Identical model path, dtype, quantization, parallelism, and revision across
source and target workers
Observed behavior
The target discovers a READY source with the same source identity, then NIXL reports an unreachable destination during remote peer setup, for example:
Destination is unreachable
UCX ERROR
NIXL_ERR_BACKEND
The target then retries/falls back. GPU memory rises from the initial model footprint to approximately two model footprints before the worker exits with OOM or exit code 137.
Summary
When ModelExpress is shared by workers in different RDMA network domains, the same model configuration can produce the same
mx_source_idfor source instances that are not mutually reachable over RDMA. A target worker can select a READY source from another domain because source selection currently matches model identity, worker rank, and accelerator compatibility, but does notconsider RDMA topology.
When the selected source is unreachable, the target falls back after the failure has already mutated the target model. The retry path reinitializes a second model while the original model is still held by the outer loader call.
The result is a near-double GPU allocation and the worker is killed by the OOM killer.
Environment
source and target workers
Observed behavior
The target discovers a READY source with the same source identity, then NIXL reports an unreachable destination during remote peer setup, for example:
The target then retries/falls back. GPU memory rises from the initial model footprint to approximately two model footprints before the worker exits with OOM or exit code 137.