fix(optimizer): robust metric resolution in slow-update longitudinal comparison - #257
Open
RohithPariki wants to merge 1 commit into
Open
fix(optimizer): robust metric resolution in slow-update longitudinal comparison#257RohithPariki wants to merge 1 commit into
RohithPariki wants to merge 1 commit into
Conversation
RohithPariki
marked this pull request as ready for review
August 26, 2026 01:35
Contributor
|
Thanks for trying to support additional metric shapes. The current fallback is too permissive for the existing rollout contract, which requires adapters to return |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
In
skillopt.optimizer.slow_update(build_comparison_pairs):Longitudinal comparison between adjacent epoch rollouts categorized per-sample outcomes (
improved,regressed,persistent_fail,stable_success) strictly usingbool(prev.get("hard", 0))andbool(curr.get("hard", 0)).When running on benchmark environments that evaluate using
score,exact_match, or floating-pointsoftthresholds without an explicithardkey in the result dictionary:bool(0) == False), misclassifying genuine improvements intopersistent_failand regressions intostable_success.0.0even when the task succeeded under primary evaluation metrics.Root Cause
Evaluation metric representations across benchmark adapters can vary (
hard,score,exact_match,soft).build_comparison_pairsassumed an explicit binary integerhardkey.Solution
_is_result_success()inskillopt/optimizer/slow_update.pyto evaluate outcome success acrosshard,score,exact_match, andsoftmetrics with robust type casting.build_comparison_pairsto use_is_result_success()for accurate categorization.tests/test_slow_update_robustness.pycovering metric categorization, field lifecycle, and serialization.Testing
pytest tests/test_slow_update_robustness.py(4/4 passed).ruff check(clean).Risk
Low. Improves classification accuracy for non-binary and custom benchmark results while preserving exact behavior for existing
hard-keyed results.