Skip to content

feat(compare): add offline result comparison - #183

Open
googs1025 wants to merge 1 commit into
alibaba:mainfrom
googs1025:feat/add-offline-result-compare
Open

feat(compare): add offline result comparison#183
googs1025 wants to merge 1 commit into
alibaba:mainfrom
googs1025:feat/add-offline-result-compare

Conversation

@googs1025

Copy link
Copy Markdown
Contributor

Summary

  • Add offline skill-up compare <old-result.json> <new-result.json> for text and JSON result diffs.
  • Compare primary case metrics, metadata, case transitions, and CI gates without rerunning evals.
  • Document the new command and update changelog.

Test Plan

  • GOCACHE=/private/tmp/skill-up-go-build-cache GOTOOLCHAIN=go1.26.4 make verify
  • GOCACHE=/private/tmp/skill-up-go-build-cache GOTOOLCHAIN=go1.26.4 make test

Closes #175

@googs1025
googs1025 requested a review from hittyt as a code owner August 12, 2026 08:05

zpzjzj commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator

Thanks for the proposal and implementation. I understand that this command provides domain-aware comparison, such as primary-case handling, case transition classification, and CI exit codes.

However, if the scope is only comparing two local result.json files, the value of adding a dedicated built-in command is not yet very clear to me. The current pass-rate, status, token, and duration comparison can already be implemented fairly cheaply with jq or a small script, while this PR introduces a relatively large amount of code and a new CLI contract that we will need to maintain.

I think the feature would be more compelling if it formed part of a complete baseline workflow—for example:

  • locating or selecting the baseline result automatically;
  • comparing against the latest release or a named baseline;
  • handling result schema compatibility across versions;
  • providing richer regression evidence beyond case status;
  • offering a reusable CI integration that individual repositories would otherwise need to maintain themselves.

Could you clarify the concrete user workflow this is intended to unlock, and what advantage the built-in command provides over a documented jq/script example? If the intention is only raw offline file comparison, I’m not yet convinced that the maintenance cost is justified.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(cli): add offline result compare command

2 participants