Skip to content

Commit 1093d78

Browse files
fix(lgtm-datamesh): retry gateway queries across KEDA scale-down races
Follow-up to 764aa3f, from a full-suite re-run: the gateway's HTTPScaledObject has scaledownPeriod 30s, so a helm rollout gate passing is no guarantee a pod is still there seconds later — three more smokes raced it: - smoke-discovery: the one-shot wake checked availableReplicas at one instant; the pod could vanish between that check and the SDL fetch. Retry the whole publish step (idempotent) with re-wake + port-forward re-attach, up to 3 attempts. - smoke-graphql: the stitched query hit a dead pinned port-forward the first time it ever ran with the HTTPScaledObject present. Same re-wake + re-attach retry. - smoke-trace-flow: goes through the interceptor by design, but the 0.12.2 interceptor's cold-start race (CAP-046) can 502 the first POSTs after a scale-from-zero. Retry the query up to 5 times. Verified live with the scaler active: smoke-graphql recovered after two dead-tunnel attempts, smoke-trace-flow after one 502; smoke-discovery passes with the guard in place. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent 764aa3f commit 1093d78

3 files changed

Lines changed: 51 additions & 11 deletions

File tree

‎examples/lgtm-datamesh/demos/smoke-discovery.sh‎

Lines changed: 21 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -110,11 +110,27 @@ sleep 3
110110

111111
# ── publish ───────────────────────────────────────────────────────────────────
112112
step "Publishing discovery contracts (OpenAPI + Protobuf + GraphQL SDL)"
113-
APICURIO_URL="http://127.0.0.1:${LOCAL_APIC}" \
114-
ORDER_URL="http://127.0.0.1:${LOCAL_ORDER}" \
115-
GATEWAY_URL="http://127.0.0.1:${LOCAL_GW}" \
116-
PROTO_PATH="$PROTO_PATH" APICURIO_GROUP="$GROUP" \
117-
./scripts/publish-discovery-contracts.sh || fail "publishing discovery contracts failed"
113+
published=0
114+
for attempt in 1 2 3; do
115+
if APICURIO_URL="http://127.0.0.1:${LOCAL_APIC}" \
116+
ORDER_URL="http://127.0.0.1:${LOCAL_ORDER}" \
117+
GATEWAY_URL="http://127.0.0.1:${LOCAL_GW}" \
118+
PROTO_PATH="$PROTO_PATH" APICURIO_GROUP="$GROUP" \
119+
./scripts/publish-discovery-contracts.sh; then
120+
published=1; break
121+
fi
122+
# The wake above is not enough on its own: the HTTPScaledObject's
123+
# scaledownPeriod is 30s, so the gateway can drop back to zero between the
124+
# rollout gate and the SDL fetch, killing the pinned port-forward mid-
125+
# publish. Re-wake, re-attach, and retry — publishing is idempotent.
126+
printf ' publish attempt %d failed — re-waking the gateway and retrying\n' "$attempt"
127+
kubectl scale deploy graphql-gateway -n "$NS" --replicas=1 >/dev/null 2>&1
128+
kubectl rollout status deployment/graphql-gateway -n "$NS" --timeout=120s >/dev/null 2>&1
129+
kill "$PF_G" 2>/dev/null
130+
kubectl port-forward -n "$NS" service/graphql-gateway "${LOCAL_GW}:80" >/dev/null 2>&1 & PF_G=$!
131+
sleep 2
132+
done
133+
(( published )) || fail "publishing discovery contracts failed"
118134

119135
# ── assert all four contract types are in the registry ────────────────────────
120136
step "Verifying contracts are registered in Apicurio"

‎examples/lgtm-datamesh/demos/smoke-graphql.sh‎

Lines changed: 18 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -106,9 +106,24 @@ printf ' order id=%s\n' "$ORDER_ID"
106106
# ── 6. query the gateway and assert stitched response ─────────────────────────
107107
step "Querying graphql-gateway for the order + nested stock (REST + gRPC stitched)"
108108
GQL_QUERY="$(printf '{"query":"{ order(id: \\"%s\\") { id itemSku quantity stock { sku quantityOnHand available } } }"}' "$ORDER_ID")"
109-
RESP="$(curl -fsS -X POST "http://127.0.0.1:${LOCAL_GQL}/graphql" \
110-
-H 'Content-Type: application/json' \
111-
-d "$GQL_QUERY")" || fail "GraphQL query failed"
109+
RESP=""
110+
for attempt in 1 2 3; do
111+
RESP="$(curl -fsS -X POST "http://127.0.0.1:${LOCAL_GQL}/graphql" \
112+
-H 'Content-Type: application/json' \
113+
-d "$GQL_QUERY")" && break
114+
# The gateway is KEDA-managed (HTTPScaledObject, scaledownPeriod 30s): with
115+
# no interceptor traffic it can scale to zero between the rollout gate and
116+
# this query, killing the pinned port-forward. Wake it, re-attach, retry.
117+
printf ' query attempt %d failed — waking the gateway and retrying\n' "$attempt"
118+
kubectl scale deploy graphql-gateway -n "$NS" --replicas=1 >/dev/null 2>&1
119+
kubectl rollout status deployment/graphql-gateway -n "$NS" --timeout=120s >/dev/null 2>&1
120+
kill "$PF_G" 2>/dev/null
121+
kubectl port-forward -n "$NS" service/graphql-gateway "${LOCAL_GQL}:80" >/dev/null 2>&1 &
122+
PF_G=$!
123+
sleep 2
124+
RESP=""
125+
done
126+
[[ -n "$RESP" ]] || fail "GraphQL query failed"
112127
printf ' %s\n' "$RESP"
113128

114129
python3 - "$RESP" "$ORDER_ID" <<'PY' || fail "GraphQL response missing stitched fields"

‎examples/lgtm-datamesh/demos/smoke-trace-flow.sh‎

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -84,9 +84,18 @@ fi
8484
# which is the downstream hop we want in the trace. With a real order the
8585
# inventory gRPC hop appears too (see smoke-graphql.sh).
8686
GQL_BODY='{"query":"{ order(id: \"trace-probe\") { id itemSku quantity stock { sku quantityOnHand available } } }"}'
87-
CODE=$(curl -s -o /dev/null -w '%{http_code}' --max-time 60 \
88-
-H "Host: $HOST" -H "Content-Type: application/json" \
89-
-X POST --data "$GQL_BODY" "http://127.0.0.1:${GQL_PORT}/graphql" || echo "000")
87+
# Retry transient non-200s: the 0.12.2 interceptor has a cold-start race
88+
# (CAP-046) where the first POSTs after a scale-from-zero can 502 while its
89+
# route to the fresh endpoint warms up.
90+
CODE="000"
91+
for attempt in 1 2 3 4 5; do
92+
CODE=$(curl -s -o /dev/null -w '%{http_code}' --max-time 60 \
93+
-H "Host: $HOST" -H "Content-Type: application/json" \
94+
-X POST --data "$GQL_BODY" "http://127.0.0.1:${GQL_PORT}/graphql" || echo "000")
95+
[[ "$CODE" == "200" ]] && break
96+
printf ' attempt %d got HTTP %s — retrying (interceptor cold-start race, CAP-046)\n' "$attempt" "$CODE"
97+
sleep 3
98+
done
9099
[[ "$CODE" == "200" ]] \
91100
|| fail "gateway did not return 200 for the GraphQL query (HTTP $CODE) — see dump"
92101
printf ' ✓ gateway processed the query (HTTP 200) — a trace should now be exporting\n'

0 commit comments

Comments
 (0)