fix(proxy): inject S3-scoped endpoint vars so s5cmd is captured too#242
Draft
christophergeyer wants to merge 1 commit into
Draft
fix(proxy): inject S3-scoped endpoint vars so s5cmd is captured too#242christophergeyer wants to merge 1 commit into
christophergeyer wants to merge 1 commit into
Conversation
The per-run S3 proxy injected only `AWS_ENDPOINT_URL`. Two problems: 1. **s5cmd bypassed the proxy.** s5cmd does not honor `AWS_ENDPOINT_URL`; it reads `--endpoint-url` / `$S3_ENDPOINT_URL`. So `roar run -- s5cmd cp s3://…` went straight to real S3, and the object was NOT recorded as a lineage input — silently losing the source (and the AI-BOM `downloadLocation` derived from it). Verified against real S3: boto3 captured, s5cmd did not, until s5cmd was pointed at the proxy explicitly. 2. **`AWS_ENDPOINT_URL` is service-agnostic** — it redirects *every* AWS service (STS, EC2, CloudWatch, …) at an S3-only proxy. A workload doing an S3 read and an `sts:AssumeRole` under `roar run` would send the STS call into the proxy. Fix: inject the S3-*scoped* SDK override `AWS_ENDPOINT_URL_S3` (honored by boto3 / aws CLI / aws-sdk-go-v2, does not touch non-S3 services) plus `S3_ENDPOINT_URL` for s5cmd. Drop the global `AWS_ENDPOINT_URL`. Upstream-endpoint detection and the `roar proxy start` hint use the same two vars. Verified end-to-end against real S3 (ModelSandbox, botocore 1.43): with no manual endpoint var, both `roar run -- python …boto3…` and `roar run -- s5cmd cp …` now record `source_type=s3` + full `source_url`; the global `AWS_ENDPOINT_URL` is unset in the child. NOTE: the Ray backend has a parallel `AWS_ENDPOINT_URL` injection (`backends/ray/submit_context.py`, `roar_worker.py`) with worker-coordination logic keyed on the var name. Left untouched here — same treatment wanted as a follow-up, but it needs the multi-node path to validate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The per-run S3 proxy injected only
AWS_ENDPOINT_URL. Two issues surfaced while validating fast S3 staging (s5cmd) for a campaign:AWS_ENDPOINT_URL— it reads--endpoint-url/$S3_ENDPOINT_URL. Soroar run -- s5cmd cp s3://…goes straight to real S3 and the object is never recorded as a lineage input — silently losing the source, and with it the AI-BOMdownloadLocationthat derives fromsource_url.AWS_ENDPOINT_URLis service-agnostic. It redirects every AWS service (STS, EC2, CloudWatch…) at an S3-only proxy. A workload that does an S3 read plus ansts:AssumeRoleunderroar runwould send the STS call into the proxy.Fix
Inject the S3-scoped SDK override
AWS_ENDPOINT_URL_S3(honored by boto3 / aws CLI / aws-sdk-go-v2; does not touch non-S3 services) plusS3_ENDPOINT_URLfor s5cmd. Drop the globalAWS_ENDPOINT_URL. Upstream-endpoint detection and theroar proxy starthint use the same two vars.Verification
Unit:
tests/unit/test_proxy_runtime_resource.pyupdated (7 pass; 19 pass across the proxy/tracer suites). ruff clean.End-to-end against real S3 (ModelSandbox, botocore 1.43), proxy enabled, no manual endpoint var set:
AWS_ENDPOINT_URL)roar run -- python …boto3… downloadAWS_ENDPOINT_URL_S3)roar run -- s5cmd cp s3://…S3_ENDPOINT_URL)Both record
source_type=s3+ fullsource_url; the child process seesAWS_ENDPOINT_URLunset (so non-S3 calls are no longer hijacked).Scope / follow-up
Deliberately limited to the non-Ray
roar runpath (proxy_resource.py) — the tested path. The Ray backend has a parallelAWS_ENDPOINT_URLinjection (backends/ray/submit_context.py,roar_worker.py) with worker-coordination logic keyed on the var name; it wants the same treatment but needs the multi-node path to validate, so it's left for a follow-up rather than shipped untested.Per direction: no backward-compat alias for the old
AWS_ENDPOINT_URL— if any existing setup relied on roar exporting the global var, it will needAWS_ENDPOINT_URL_S3instead.🤖 Generated with Claude Code