Bandwidth-aware Run:ai streamer tuning + S3 Gateway VPC endpoint - #91
Merged
Conversation
Every benchmark provisions a fresh GPU node that pulls model weights (Run:ai streamer from S3) and S3-backed ECR image layers. Without an S3 Gateway endpoint that traffic egresses through the NAT gateway — a shared bandwidth ceiling + per-GB data-processing charge, and a bottleneck when multiple nodes pull at once (AWS fast-model-loading guidance flags exactly this). Add a Gateway VPC endpoint for S3 (free) that injects the S3 prefix-list route into the private route tables, so S3 traffic stays on the AWS network. Wired in the vpc module (new region var for the service name; route_table_ids from the module's private_route_table_ids). Greenfield only — this module isn't created in brownfield mode. terraform validate green. (Not applied — infra is currently destroyed; lands on the next greenfield apply.) Signed-off-by: Jeremy Cowan <jicowan@hotmail.com>
Tune the Run:ai model streamer per instance network bandwidth, reconciling
AWS's EKS "fast model loading" guide with run.ai's upstream docs — the two
give different but internally-consistent profiles because the optimum
depends on the pipe:
- distributed:true in --model-loader-extra-config when TensorParallelDegree
> 1 (both sources agree). Each vLLM rank streams its own shard over
torch-distributed instead of rank-0 broadcasting; requires vLLM V1,
incompatible with --enforce-eager. Omitted at TP=1. Wired through
single-node, D/P (per-role TP), and the manifest export via one shared
streamerExtraConfig so deploy and export can't drift.
- Bandwidth-aware chunk + concurrency (internal/runtime/instance_network.go,
sourced from AWS EC2 accelerated-computing network specs):
* High-bandwidth instances (>=50 Gbps: g*.24xl/48xl, g6e.12xl+, g7e,
p4/p5) get AWS's profile — RUNAI_STREAMER_CHUNK_BYTESIZE=4 GiB +
size-derived concurrency ceil(model_gb / 4), capped at 64.
* Standard instances (the small/bursty G sizes, 2.5-40 Gbps) get run.ai's
profile — no chunk override (8 MiB object-storage default) + the
benchmarked flat concurrency 32.
An explicit operator StreamerConcurrency overrides either profile.
- RUNAI_STREAMER_S3_REQUEST_TIMEOUT_MS=3000 + S3_LOW_SPEED_LIMIT=1048576 on
every streamed load. Both are real (present in the streamer's C++ source,
just undocumented in its markdown; upstream defaults 1000 ms / disabled) —
they abort+retry a stalled S3 request and don't depend on bandwidth.
Rationale confirmed against the run.ai C++ source and AWS's per-instance
network specs rather than the markdown docs, which lag the code and are
hardware-specific. PP (co-located) never streams today, so distributed's
known TP+PP multi-group hang (run.ai #165) isn't reachable.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Jeremy Cowan <jicowan@hotmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Tunes the Run:ai model streamer per instance network bandwidth and adds an S3 Gateway VPC endpoint to keep model/image pulls off the NAT gateway.
internal/runtime/instance_network.go): high-bandwidth instances (≥50 Gbps — g*.24xl/48xl, g6e.12xl+, g7e, p4/p5) get AWS's profile (4 GiBRUNAI_STREAMER_CHUNK_BYTESIZE+ size-derived concurrencyceil(model_gb/4), capped 64); standard instances get run.ai's profile (no chunk override, flat concurrency 32). Explicitstreamer_concurrencyoverrides either.distributed:truein--model-loader-extra-configwhen TP>1, gated off at TP=1. Wired through single-node, D/P, and manifest export via one sharedstreamerExtraConfigso deploy and export can't drift.RUNAI_STREAMER_S3_REQUEST_TIMEOUT_MS=3000,RUNAI_STREAMER_S3_LOW_SPEED_LIMIT=1048576) on every streamed load — both real (present in the streamer's C++ source).Rationale confirmed against the run.ai C++ source and AWS per-instance network specs, not the markdown docs (which lag).
Live validation (2026-08-10)
Provisioned a fresh
accelbench-eksand ran an end-to-end benchmark streaming a cached S3 model (Qwen2.5-1.5B on g6.2xlarge, a standard-bandwidth instance):--load-format runai_streamer+"concurrency":32(run.ai standard-BW profile, no 4 GiB chunk env) + the S3 retry envs +memory_limit; nodistributedat TP=1 — exactly as designed.Loading safetensors using Runai Model Streamer: 100% Completed 338/338 in 3s.go test ./...+go vet ./...green. No frontend changes.🤖 Generated with Claude Code