Repository navigation
ci(ai-evals): Send each repeat's results into its own run - #387
Merged
Merged
Conversation
Use the job's run-id from create-runs, falling back to the label and side's run, so :x<count> labels keep working once sentry-for-ai-evals makes a run per repeat.
jaydgoss
added
the
ai-evals:llms_txt:sonnet-5.5
Sentry Evalssentry-for-ai-evals:llms_txt:sonnet-5.5 (Comparison)
|
1 of 6 tasks
jaydgoss
added a commit
to getsentry/sentry-docs
that referenced
this pull request
Oct 30, 2026
## DESCRIBE YOUR PR Adds a `report` job to the `AI Evals` workflow that posts each label's products matrix (test x target x product, base and head) to Slack once the eval jobs finish, using the `post-matrix` action from [sentry-for-ai-evals#111](getsentry/sentry-for-ai-evals#111). This repo is public, so the matrix can't go on the PR. [VDY-309](https://linear.app/getsentry/issue/VDY-309/onboarding-evals-post-a-pr-runs-products-matrix-to-slack) - Same job and condition as [getsentry/sentry-for-ai#385](getsentry/sentry-for-ai#385) and [sentry-for-ai-evals#115](getsentry/sentry-for-ai-evals#115). It runs only when the eval jobs succeeded or failed, so not when a new push cancelled them, when there's no preview, or for fork PRs. - The log shows only whether each label was posted. The channel is set in sentry-for-ai-evals, so this repo doesn't name it. - Posts with the `EVALS_SLACK_BOT_TOKEN` secret, which is set on this repo. Without it, the job logs a notice and posts nothing. - Also sends each eval job's results into its own run (`matrix.job.run-id`, falling back to the label and side's run), so `:x<count>` labels keep working once [sentry-for-ai-evals#137](getsentry/sentry-for-ai-evals#137) makes a run per repeat. Same change as [getsentry/sentry-for-ai#387](getsentry/sentry-for-ai#387). [VDY-327](https://linear.app/getsentry/issue/VDY-327/onboarding-evals-upload-a-run-per-repeat-with-a-01-score-per-product) Testing: `ai-evals:fastapi:sonnet-5.5` on this PR posted its matrix to Slack ([run](https://github.com/getsentry/sentry-docs/actions/runs/40583152243)). ## IS YOUR CHANGE URGENT? Help us prioritize incoming PRs by letting us know when the change needs to go live. Select exactly one option. For deadlines, replace `YYYY-MM-DD` with the due date. You can update this information later by editing the PR description. - [ ] Urgent deadline (GA date, etc.): YYYY-MM-DD - [x] Other deadline: 2026-10-30 - [ ] No deadline: Not urgent, can wait up to 1 week+ ## SLA - Teamwork makes the dream work, so please add a reviewer to your PRs. - Please give the docs team up to 1 week to review your PR unless you've supplied a deadline. Thanks in advance for your help! ## PRE-MERGE CHECKLIST _Make sure you've checked the following before merging your changes:_ - [ ] Checked Vercel preview for correctness, including links - [ ] PR was reviewed and approved by any necessary SMEs (subject matter experts) - [ ] PR was reviewed and approved by a member of the [Sentry docs team](https://github.com/orgs/getsentry/teams/docs) 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Once getsentry/sentry-for-ai-evals#137 lands,
create-runsmakes a run per repeat of an:x<count>label and puts each job's run inmatrix.job.run-id. The eval jobs here still look up the label and side's run, so every repeat would send into the repeat-1 run, overwriting each other (same scenario names), and the other repeat runs would stay pending until the 30-minute timeout.matrix.job.run-id, falling back to the old lookup, so this works before and after getsentry/sentry-for-ai-evals#137 merges.Testing:
ai-evals:llms_txt:sonnet-5.5on this PR (the fallback path, since getsentry/sentry-for-ai-evals#137 isn't merged).🤖 Generated with Claude Code