feat(trackio): capture experiments + record the hosted URL on the DAG ([70]) - #247
Merged
Conversation
…RL on the DAG ([70]) Two pieces, both zero-metric-storage (TReqs stores no customer data — the DAG carries a URL to the externally-hosted experiment, like a W&B URL): 1. Shim injection (roar run --capture-trackio [--trackio-space OWNER/NAME]): the injected sitecustomize aliases the workload's wandb -> trackio and installs a minimal mlflow -> trackio shim, patching trackio.init to sync to the HF Space. So an UNMODIFIED repo's W&B/MLflow logging funnels into a hosted trackio experiment with no code edit. Gated by ROAR_CAPTURE_TRACKIO; best-effort (a missing trackio is a silent no-op). Extracted to trackio_shim.py for testing. (Empirically grounded end-to-end: the same shim captured real wandb+mlflow runs, and trackio.init(space_id=...) yields a live public HF-Space dashboard.) 2. ExperimentTrackerAnalyzer: new _extract_trackio_info reads trackio's per-project SQLite for run identity only and emits the hosted Space URL (https://huggingface.co/spaces/<space>?project=<project>) onto the DAG. URL-only — never the metrics. space_id from ROAR_TRACKIO_SPACE_ID or the synced run. Tests: 5 analyzer (incl. a no-metric-leak invariant) + 5 shim = 10; system_labels regression green; ruff + mypy clean. Should ride the 0.4.1 release. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…crape-only, no flags Corrected per Chris: roar does NOTHING trackio-specific at run time (no shim, no --capture-trackio/--trackio-space flags, no env). It just SCRAPES the hosted- experiment link trackio leaves behind — exactly like it scrapes the W&B URL. _extract_trackio_info reads the run identity + the Space from trackio's own project_metadata (key='space_id', where trackio persists it after a run syncs) and emits https://huggingface.co/spaces/<space>?project=<project>. URL-only — never the metrics (TReqs stores no customer data). Grounded against a real synced run. The wandb->trackio aliasing (making an unmodified wandb repo emit trackio) is a separate, campaign-side concern for reproducibleailist.org — NOT roar, NOT a CLI arg — so the shim + flags added earlier are removed here. Tests: 4 analyzer (project_metadata scrape + no-space + end-to-end + no-metric-leak); ruff + mypy clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
christophergeyer
marked this pull request as ready for review
July 27, 2026 16:26
Fixes two QA-found bugs in the trackio experiment scraper: - F1 (cross-DB contamination): the extractor reused a single shared dict across all *.db files, so a run touching >1 trackio project could splice one project's Space onto another's name (.../alpha-space?project=zebra). Now emits one record per project DB and builds each URL per-DB. - F2 (schema brittleness): the run-identity and space-id lookups shared one try-block and hardcoded SELECT run_id, run_name, so an older/imported trackio schema (no run_id column, or no project_metadata table) discarded BOTH the identity and the URL. The two lookups are now independent and the run-identity query adapts to a run_id-less metrics table. Metric-leak invariant preserved (only run_id/run_name read, never metrics). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jul 27, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Standardize on trackio as the experiment sink, with zero repo code change, and record the hosted dashboard URL on the DAG.
1. Shim injection —
roar run --capture-trackio [--trackio-space reproducible-ai/experiments]. The injected sitecustomize aliases the workload'swandb→ trackio and installs a minimalmlflow→ trackio shim (patchingtrackio.initto sync to the HF Space). An unmodified repo's W&B/MLflow logging funnels into a hosted trackio experiment. Gated byROAR_CAPTURE_TRACKIO; best-effort (missing trackio = silent no-op). Also settable via env /.roar/config.toml [env].2. URL capture —
ExperimentTrackerAnalyzer._extract_trackio_inforeads trackio's per-project SQLite for run identity only and emits the public Space URL (…/spaces/<space>?project=<project>) onto the DAG. URL-only — never the metrics (TReqs stores no customer data; the metrics live on the hosted dashboard, like a W&B URL).Grounding
The shim + hosting were proven end-to-end before this PR: the same wandb→trackio + mlflow→trackio shims captured real runs (verified in trackio's SQLite), and
trackio.init(space_id=…)produced a live public HF-Space dashboard on our org.Tests
5 analyzer (incl. a no-metric-leak invariant + fake trackio SQLite fixture) + 5 shim (install/gating/space-injection/no-clobber/missing-trackio) = 10;
test_system_labelsregression green; ruff + mypy clean.Sibling of #245 (
get --cache) / #246 (put → HF). Should ride 0.4.1.🤖 Generated with Claude Code