feat(get): --cache fetches from a mirror, records the canonical source ([17]) - #245
Merged
Conversation
A roar get'd file becomes a SOURCE node, but its artifact row carried NULL source_type/source_url — the URL lived only in the get job's command. So a downstream AI-BOM had no source to derive downloadLocation from (the confirmed 97.5-not-100 gap for http/get-fetched data). The recorder and schema already support these fields (schema comment: source_type 'https', NULL=local); the get service just never set them. Set source_type = parsed scheme and source_url = each file's canonical remote_url (falling back to the request source). Adds a GetRequest.cache field: when a mirror is used, the CANONICAL source is still recorded, so downloadLocation stays the public URL, not the private mirror. (The --cache FETCH path is a follow-up commit.) Repro fixed end-to-end: roar get https://... now records source_type=https + source_url on the artifact (was NULL). Regression tests added. ruff+mypy clean, 36 get tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds `roar get <url> --cache <mirror>`: fetch bytes from the mirror (the in-region cost win) while recording <url> as the canonical source_type/source_url (so the AI-BOM downloadLocation stays the public URL, never the private mirror). The mirror is hash-verified against --hash when given, and by content-addressing at reproduce time; on miss / unreachable / hash-mismatch it falls back to the canonical source. The recorded command carries --cache so a reproduce replays cache-first-then-canonical. Verified end-to-end: fetch-from-mirror-record-canonical, and fallback on an unreachable mirror. ruff+mypy clean, 26 get tests pass (4 new). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
roar get <canonical-url> --cache <s3-url>— the DAG-source design settled in the campaign (infra-spec §3):source_url/downloadLocation(stranger-reachable), never the cache.--cacheis a declared accelerator — fetch prefers the (fast, private) mirror, hash-verifies the bytes against the canonical expectation, and falls back to canonical on any cache miss/corruption. Cache is trusted for speed, never identity.--cachearg is recorded on the get step and propagated toreproduce, so capture and reproduce behave identically.source_type/source_urlonget'd artifacts (complements fix(register): carry source_url into artifact metadata for AI-BOM downloadLocation #239's registration-side carry).Why
Lets expensive instances pre-fetch datasets from a fast S3 mirror without polluting the DAG's provenance — the recorded origin stays the public canonical URL. Output-side sibling is [71] (
roar put→ HF).Tests
tests/application/get/test_service.py— 10 passing (cache-hit/verify/fallback, canonical recording). ruff + format clean (pinned 0.15.21). Merged rc/0.4.1 in (ruff pin).Should ride the 0.4.1 release.
🤖 Generated with Claude Code