feat(llm): add multi-replica worker routing

.github/workflows/bazel.yml

@@ -88,6 +88,7 @@ jobs:
         run: |
           bash tools/ci/test-bazel-cache-upload-mode
           bash tools/ci/test-bazel-remote-probe
+          bash tools/ci/test-image-push-manual
 
       - name: Compute changed subtrees
         id: detect

.github/workflows/image-push-manual.yml

@@ -167,6 +167,8 @@ jobs:
             exit 1
           fi
           echo "discovered: ${indexes[*]}"
+          declare -A seen_repos=()
+          repos=()
           for tgt in "${indexes[@]}"; do
             name="${tgt##*:}"; name="${name%_index}"
             # Two naming conventions exist in the tree and they mean different
@@ -186,6 +188,20 @@ jobs:
               *-image) repo="${name%-image}" ;;
               *)       sub="$(printf '%s' "$name" | tr '_' '-')"; repo="${svc}-${sub}" ;;
             esac
+            if [ -n "${seen_repos[$repo]:-}" ]; then
+              echo "ERROR: ${tgt} and ${seen_repos[$repo]} both map to image repository ${repo}" >&2
+              echo "Give each oci_image_index target a distinct image name before publishing." >&2
+              exit 1
+            fi
+            seen_repos["$repo"]="$tgt"
+            repos+=("$repo")
+          done
+          # Validate every target-to-repository mapping before starting any
+          # push. A later collision must not leave earlier repositories with
+          # a new snapshot or latest-dispatch tag.
+          for i in "${!indexes[@]}"; do
+            tgt="${indexes[$i]}"
+            repo="${repos[$i]}"
             dest="${REGISTRY}/${repo}"
             echo "[push] ${tgt} -> ${dest}:${TAG} (+ latest-dispatch)"
             mkdir -p ci-ghcr

deploy/helm/gateway-routes/README.md

@@ -4,17 +4,26 @@ This repository contains the Helm chart for deploying NVCF ingress routes via th
 
 ## Overview
 
-The chart deploys `HTTPRoute`, `TCPRoute`, and `ReferenceGrant` resources that attach to an existing Gateway provisioned separately by the cluster operator (e.g. Envoy Gateway, Istio, Traefik, Kong). It also includes optional `PodMonitor` resources for scraping Envoy Gateway proxy metrics with Prometheus.
-
-The chart deploys routing configuration only. It does not include any container images. Backend services referenced by the routes (`api`, `nvct-api`, `api-keys`, `invocation`, `llm-api-gateway`, `vanity-gateway`, `reval`, `sis`, `grpc`, `nats`) must already be deployed separately.
+The chart deploys `HTTPRoute`, `GRPCRoute`, `TCPRoute`, `UDPRoute`, and
+`ReferenceGrant` resources that attach to an existing Gateway provisioned
+separately by the cluster operator, such as Envoy Gateway, Istio, Traefik, or
+Kong. It also includes optional `PodMonitor` resources for scraping Envoy
+Gateway proxy metrics with Prometheus.
+
+The chart deploys routing configuration only. It does not include any
+container images. Backend services referenced by the routes (`api`,
+`nvct-api`, `api-keys`, `invocation`, `llm-api-gateway`,
+`llm-request-router-backend-router`, `vanity-gateway`, `reval`, `sis`, `grpc`,
+`nats`) must already be deployed separately.
 
 ## Prerequisites
 
 - Kubernetes cluster
 - Helm 3.x
 - `kubectl`
 - A Gateway API compatible controller installed in the cluster
-- An existing `Gateway` resource with an HTTP listener (and TCP listeners if the gRPC or NATS routes are enabled)
+- Existing `Gateway` resources with the listeners required by each enabled route
+- A Gateway controller with `UDPRoute` support when LLM worker routing is enabled
 - The backend services that the routes target, deployed in their respective namespaces
 
 ## Getting Started
@@ -57,6 +66,8 @@ Important settings to review before deployment:
 - `nvcfGatewayRoutes.gateways.shared.*` for the HTTP Gateway name, namespace, and listener
 - `nvcfGatewayRoutes.gateways.grpc.*` for the TCP Gateway name, namespace, and listener
 - `nvcfGatewayRoutes.gateways.nats.*` for the NATS TCP Gateway name, namespace, and listener
+- `nvcfGatewayRoutes.gateways.llmGrpc.*` for the LLM worker gRPC TCP listener
+- `nvcfGatewayRoutes.gateways.llmQuic.*` for the LLM reverse-tunnel UDP listener
 - `nvcfGatewayRoutes.routes.<route>.enabled` to toggle individual routes
 - `nvcfGatewayRoutes.routes.nvcfApi.grpc.enabled` and
   `nvcfGatewayRoutes.routes.nvctApi.grpc.enabled` to expose API gRPC routes
@@ -87,6 +98,7 @@ Enabled `HTTPRoute` entries must not share a resolved hostname because each `HTT
 | `grpc` | TCPRoute | Not rendered | `grpc.nvcf:10081` |
 | `grpcWorker` | TCPRoute (disabled by default) | Not rendered | `grpc.nvcf:10086` |
 | `nats` | TCPRoute (disabled by default) | Not rendered | `nats.nats-system:4222` |
+| `llmWorker` | TCPRoute and UDPRoute (disabled by default) | Not rendered | `llm-request-router-backend-router.<backend namespace>:50071/TCP,50072/UDP` |
 
 Cross-namespace routing is supported via `ReferenceGrant` resources rendered into each backend namespace.
 
@@ -97,3 +109,9 @@ Cross-namespace routing is supported via `ReferenceGrant` resources rendered int
 - The `grpc` TCPRoute does not enforce HTTP hostname matching at the Gateway layer. Configure DNS or TCP load balancer routing outside this chart.
 - The `grpcWorker` TCPRoute is beta support for split or multi-cluster gRPC worker callbacks. It carries HTTP/1 CONNECT callback traffic only. Enable it only when the control-plane grpc-proxy runs one replica with HPA disabled. Multi-replica grpc-proxy requires pod-specific callback routing and is not supported by this shared TCPRoute.
 - Enabling the `nats` route requires a reachable TCP listener for NATS on the referenced Gateway. The HTTP Gateway address does not imply NATS reachability unless that same Gateway also has the NATS TCP listener configured.
+- The `llmWorker` routes target Stargate's authority/SNI-aware backend router.
+  Set `nvcfGatewayRoutes.routes.llmWorker.backend.namespace` to the effective
+  namespace of the `llm-request-router` release. The gateway chart cannot
+  derive the namespace of a separate Helm release.
+  Keep the TCP and UDP Gateways separate when the infrastructure requires
+  separate load balancers for each protocol.

deploy/helm/gateway-routes/chart/templates/_helpers.tpl

@@ -53,6 +53,10 @@ app.kubernetes.io/name: {{ include "nvcf-gateway.name" . }}
 app.kubernetes.io/instance: {{ .Release.Name }}
 {{- end }}
 
+{{- define "nvcf-gateway.llmWorkerBackendNamespace" -}}
+{{- required "nvcfGatewayRoutes.routes.llmWorker.backend.namespace is required when llmWorker.enabled is true" .Values.nvcfGatewayRoutes.routes.llmWorker.backend.namespace -}}
+{{- end }}
+
 {{/*
 Validate that enabled HTTPRoutes do not compete for the same hostname and
 root PathPrefix match on the shared Gateway. All HTTPRoute templates in this

deploy/helm/gateway-routes/chart/templates/referencegrant-llm-worker.yaml

@@ -0,0 +1,25 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+{{- if and .Values.nvcfGatewayRoutes.enabled .Values.nvcfGatewayRoutes.routes.llmWorker.enabled }}
+{{- $backendNamespace := include "nvcf-gateway.llmWorkerBackendNamespace" . }}
+apiVersion: gateway.networking.k8s.io/v1beta1
+kind: ReferenceGrant
+metadata:
+  name: allow-llm-worker-routes
+  namespace: {{ $backendNamespace }}
+  labels:
+    {{- include "nvcf-gateway.labels" . | nindent 4 }}
+spec:
+  from:
+    - group: gateway.networking.k8s.io
+      kind: TCPRoute
+      namespace: {{ .Values.nvcfGatewayRoutes.gateways.llmGrpc.namespace }}
+    - group: gateway.networking.k8s.io
+      kind: UDPRoute
+      namespace: {{ .Values.nvcfGatewayRoutes.gateways.llmQuic.namespace }}
+  to:
+    - group: ""
+      kind: Service
+      name: {{ .Values.nvcfGatewayRoutes.routes.llmWorker.backend.name }}
+{{- end }}

deploy/helm/gateway-routes/chart/templates/tcproute-llm-worker.yaml

@@ -0,0 +1,28 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+{{- if and .Values.nvcfGatewayRoutes.enabled .Values.nvcfGatewayRoutes.routes.llmWorker.enabled }}
+{{- $backendNamespace := include "nvcf-gateway.llmWorkerBackendNamespace" . }}
+apiVersion: gateway.networking.k8s.io/v1alpha2
+kind: TCPRoute
+metadata:
+  name: {{ .Values.nvcfGatewayRoutes.routes.llmWorker.name }}-grpc
+  namespace: {{ .Values.nvcfGatewayRoutes.gateways.llmGrpc.namespace }}
+  labels:
+    {{- include "nvcf-gateway.labels" . | nindent 4 }}
+    app.kubernetes.io/component: llm-worker-grpc-route
+  {{- with .Values.nvcfGatewayRoutes.routes.llmWorker.routeAnnotations }}
+  annotations:
+    {{- toYaml . | nindent 4 }}
+  {{- end }}
+spec:
+  parentRefs:
+    - name: {{ .Values.nvcfGatewayRoutes.gateways.llmGrpc.name }}
+      namespace: {{ .Values.nvcfGatewayRoutes.gateways.llmGrpc.namespace }}
+      sectionName: {{ .Values.nvcfGatewayRoutes.gateways.llmGrpc.listenerName }}
+  rules:
+    - backendRefs:
+        - name: {{ .Values.nvcfGatewayRoutes.routes.llmWorker.backend.name }}
+          namespace: {{ $backendNamespace }}
+          port: {{ .Values.nvcfGatewayRoutes.routes.llmWorker.backend.grpcPort }}
+{{- end }}

deploy/helm/gateway-routes/chart/templates/udproute-llm-worker.yaml

@@ -0,0 +1,28 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+{{- if and .Values.nvcfGatewayRoutes.enabled .Values.nvcfGatewayRoutes.routes.llmWorker.enabled }}
+{{- $backendNamespace := include "nvcf-gateway.llmWorkerBackendNamespace" . }}
+apiVersion: gateway.networking.k8s.io/v1alpha2
+kind: UDPRoute
+metadata:
+  name: {{ .Values.nvcfGatewayRoutes.routes.llmWorker.name }}-quic
+  namespace: {{ .Values.nvcfGatewayRoutes.gateways.llmQuic.namespace }}
+  labels:
+    {{- include "nvcf-gateway.labels" . | nindent 4 }}
+    app.kubernetes.io/component: llm-worker-quic-route
+  {{- with .Values.nvcfGatewayRoutes.routes.llmWorker.routeAnnotations }}
+  annotations:
+    {{- toYaml . | nindent 4 }}
+  {{- end }}
+spec:
+  parentRefs:
+    - name: {{ .Values.nvcfGatewayRoutes.gateways.llmQuic.name }}
+      namespace: {{ .Values.nvcfGatewayRoutes.gateways.llmQuic.namespace }}
+      sectionName: {{ .Values.nvcfGatewayRoutes.gateways.llmQuic.listenerName }}
+  rules:
+    - backendRefs:
+        - name: {{ .Values.nvcfGatewayRoutes.routes.llmWorker.backend.name }}
+          namespace: {{ $backendNamespace }}
+          port: {{ .Values.nvcfGatewayRoutes.routes.llmWorker.backend.quicPort }}
+{{- end }}

deploy/helm/gateway-routes/chart/values.yaml

@@ -51,6 +51,17 @@ nvcfGatewayRoutes:
       namespace: gateway
       # Listener (sectionName) on the Gateway to attach the route to
       listenerName: nats
+    # TCP Gateway for LLM worker registration and Stargate watches.
+    llmGrpc:
+      name: llm-grpc-gateway
+      namespace: gateway
+      listenerName: llm-grpc
+    # UDP Gateway for LLM reverse tunnels. This may be a separate load
+    # balancer from the TCP Gateway.
+    llmQuic:
+      name: llm-quic-gateway
+      namespace: gateway
+      listenerName: llm-quic
 
   # HTTPRoute configurations
   routes:
@@ -239,6 +250,21 @@ nvcfGatewayRoutes:
         namespace: nats-system
         port: 4222
       routeAnnotations: {}
+
+    # Backend-facing LLM registration/watch and reverse-tunnel routes.
+    # The backend router selects the correct Stargate pod by gRPC authority
+    # and QUIC SNI, so this route supports multi-replica request routers.
+    llmWorker:
+      enabled: false
+      name: llm-worker
+      backend:
+        name: llm-request-router-backend-router
+        # Required when llmWorker is enabled. Set this to the effective
+        # llm-request-router namespace.
+        namespace: ""
+        grpcPort: 50071
+        quicPort: 50072
+      routeAnnotations: {}
 
     # NVCF Worker Container needs to be able to fetch secrets for function pods
     # In split/multi cluster planes, ess resides in control-plane and requires

deploy/helm/gateway-routes/scripts/check-llm-worker-routes.sh

@@ -0,0 +1,81 @@
+#!/usr/bin/env bash
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+set -euo pipefail
+
+script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+chart_dir="${script_dir}/../chart"
+rendered="$(mktemp)"
+disabled="$(mktemp)"
+trap 'rm -f "$rendered" "$disabled"' EXIT
+
+helm template nvcf-gateway-routes "$chart_dir" \
+  --namespace gateway \
+  --set nvcfGatewayRoutes.routes.llmWorker.enabled=true \
+  --set nvcfGatewayRoutes.gateways.llmGrpc.name=llm-grpc-gateway \
+  --set nvcfGatewayRoutes.gateways.llmGrpc.namespace=gateway \
+  --set nvcfGatewayRoutes.gateways.llmQuic.name=llm-quic-gateway \
+  --set nvcfGatewayRoutes.gateways.llmQuic.namespace=gateway \
+  --set nvcfGatewayRoutes.routes.llmWorker.backend.namespace=router-system \
+  >"$rendered"
+
+assert_contains() {
+  local pattern="$1"
+  local message="$2"
+  if ! grep -Fq -- "$pattern" "$rendered"; then
+    echo "FAIL: ${message}" >&2
+    exit 1
+  fi
+}
+
+assert_contains "kind: TCPRoute" \
+  "LLM worker routing must expose gRPC registration over TCP"
+assert_contains "kind: UDPRoute" \
+  "LLM worker routing must expose reverse tunnels over UDP"
+assert_contains "name: llm-request-router-backend-router" \
+  "LLM worker routes must target the authority/SNI-aware backend router"
+assert_contains "name: allow-llm-worker-routes" \
+  "ReferenceGrant must permit cross-namespace LLM worker routes"
+assert_contains "sectionName: llm-grpc" \
+  "TCPRoute must attach to the configured LLM gRPC listener"
+assert_contains "sectionName: llm-quic" \
+  "UDPRoute must attach to the configured LLM QUIC listener"
+
+reference_grant_service_name="$(awk '
+  $0 == "kind: ReferenceGrant" { in_grant = 1; target_grant = 0; in_to = 0 }
+  in_grant && !target_grant && $1 == "name:" && $2 == "allow-llm-worker-routes" { target_grant = 1 }
+  target_grant && $0 == "  to:" { in_to = 1 }
+  target_grant && in_to && $1 == "name:" { print $2; exit }
+' "$rendered")"
+if [[ "$reference_grant_service_name" != "llm-request-router-backend-router" ]]; then
+  echo "FAIL: LLM worker ReferenceGrant must stay scoped to the configured backend Service" >&2
+  exit 1
+fi
+
+backend_namespace_references="$(grep -Fc -- "namespace: router-system" "$rendered")"
+if [[ "$backend_namespace_references" != "3" ]]; then
+  echo "FAIL: LLM worker routes and ReferenceGrant must use the configured backend namespace" >&2
+  exit 1
+fi
+
+if helm template nvcf-gateway-routes "$chart_dir" \
+  --namespace gateway \
+  --set nvcfGatewayRoutes.routes.llmWorker.enabled=true \
+  --set-string nvcfGatewayRoutes.routes.llmWorker.backend.namespace= \
+  >/dev/null 2>&1; then
+  echo "FAIL: enabled LLM worker routing must require an explicit backend namespace" >&2
+  exit 1
+fi
+
+helm template nvcf-gateway-routes "$chart_dir" \
+  --namespace gateway \
+  --set nvcfGatewayRoutes.routes.llmWorker.enabled=false \
+  >"$disabled"
+
+if grep -Eq '^  name: (llm-worker-(grpc|quic)|allow-llm-worker-routes)$' "$disabled"; then
+  echo "FAIL: disabled LLM worker routing must not render route resources" >&2
+  exit 1
+fi
+
+echo "PASS: LLM worker Gateway routes render correctly"

deploy/helm/llm-request-router/README.md

@@ -4,7 +4,17 @@ This repository contains the Helm chart for deploying the NVCF LLM Request Route
 
 ## Overview
 
-The chart packages the LLM Request Router StatefulSet with HTTP and gRPC services, a metrics endpoint, and a headless service for multi-instance DNS discovery. A Vault Agent sidecar is configured to fetch a service token from a Vault or OpenBao backend; the application reads `nvcfApiToken` from `/vault/secrets/secrets.json` and attaches it as a Bearer token to outgoing worker authentication gRPC calls.
+The chart packages the LLM Request Router StatefulSet with HTTP and gRPC
+services, a metrics endpoint, and a headless service for multi-instance DNS
+discovery. It can also deploy the Stargate Kubernetes backend router for
+worker gRPC registration and reverse QUIC tunnels through a shared Gateway or
+load balancer. The backend router selects the correct Stargate pod from gRPC
+authority and QUIC SNI.
+
+A Vault Agent sidecar is configured to fetch a service token from a Vault or
+OpenBao backend. The application reads `nvcfApiToken` from
+`/vault/secrets/secrets.json` and attaches it as a Bearer token to outgoing
+worker authentication gRPC calls.
 
 The default chart values do not set the required image registry and repository. They must be supplied through an additional values file at install time, and access to those images must be arranged separately.
 
@@ -70,6 +80,7 @@ Important settings to review before deployment:
 - `llmRequestRouter.imagePullSecrets` for private registry access
 - `llmRequestRouter.replicaCount`, resource requests, and limits for your environment
 - `llmRequestRouter.service.*` for HTTP, gRPC, metrics, and headless service ports
+- `llmRequestRouter.backendRouter.*` for multi-replica worker gRPC and reverse-tunnel routing
 - `llmRequestRouter.metrics.enabled` to expose the metrics port on the Service (default: `false`)
 - `llmRequestRouter.metrics.serviceMonitor.enabled` to create a Prometheus `ServiceMonitor` (requires `metrics.enabled`)
 - `llmRequestRouter.certificate.*` to let cert-manager issue the Stargate QUIC server certificate
@@ -80,6 +91,51 @@ Important settings to review before deployment:
 
 The default values include development-oriented placeholders. Override them before using the chart in any shared or production environment.
 
+## Backend Worker Routing
+
+Enable `llmRequestRouter.backendRouter.enabled` when workers reach a
+multi-replica request router through a shared endpoint. Set both pylon dial
+addresses to the external endpoints that workers can resolve:
+
+```yaml
+llmRequestRouter:
+  backendRouter:
+    enabled: true
+    image:
+      tag: <published-stargate-version>
+    pylonGrpcDialAddress: llm-router.example.com:443
+    pylonReverseTunnelDialAddress: llm-router.example.com:8080
+```
+
+The chart uses the main Stargate image for both workloads. The image must
+contain `/usr/local/bin/stargate-k8s-router`. Set
+`llmRequestRouter.backendRouter.image.tag` to an image version that contains
+that binary. The chart requires this explicit pin when backend routing is
+enabled.
+
+The backend router watches EndpointSlices. The chart creates a dedicated
+ServiceAccount by default and binds a namespaced Role to it when
+`llmRequestRouter.rbac.create=true`. When
+`llmRequestRouter.backendRouter.serviceAccount.create=false`, set
+`llmRequestRouter.backendRouter.serviceAccount.name` to an existing account.
+When `rbac.create=false`, grant `get`, `list`, and `watch` on
+`discovery.k8s.io/endpointslices` to that account outside this chart.
+
+Route TCP port `50071` and UDP port `50072` to the
+`llm-request-router-backend-router` Service. The NVCF gateway-routes chart can
+create the matching `TCPRoute`, `UDPRoute`, and `ReferenceGrant` resources.
+The Gateway implementation must support Gateway API `UDPRoute`.
+
+When QUIC verification is enabled, the mounted certificate must cover the
+worker-facing reverse-tunnel hostname and the per-pod hostname template. The
+default template is
+`{pod_name}.llm-request-router-headless.<namespace>.svc.cluster.local`.
+
+Stargate and the backend router read the TLS certificate and key only during
+process startup. After cert-manager or another issuer renews the Secret,
+restart both workloads or configure a Secret reloader that triggers their
+rollouts.
+
 ## Load Balancer Configuration
 
 The chart can pass a Stargate load-balancer config in either of two ways: