Skip to content

Fixes and improvements to evaluation parsing#63

Open
tzschmidt wants to merge 9 commits into
masterfrom
bugfixes
Open

Fixes and improvements to evaluation parsing#63
tzschmidt wants to merge 9 commits into
masterfrom
bugfixes

Conversation

@tzschmidt
Copy link
Copy Markdown
Collaborator

  • Update evaluation format
  • Update evaluation parser
  • Improve documentation

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the benchmark evaluation XML format and aligns both the writer (runscript) and reader (result parser/classes) to the new structure, with accompanying test and documentation updates.

Changes:

  • Extend evaluation/result model to capture richer metadata (e.g., cmdline as {pre, post}, memout, template_options, dist_template/dist_options, per-setting encodings).
  • Refactor the result XML parser into smaller helpers with stricter required-attribute handling and clearer errors.
  • Update tests, reference XML, and getting-started docs to reflect the new attributes/paths.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
src/benchmarktool/result/parser.py Refactors XML parsing into helper methods; parses new attributes (cmdline dicts, memout, template options, setting encodings).
src/benchmarktool/result/result.py Updates dataclasses (new fields, kw_only=True in several types) to represent the new parsed format.
src/benchmarktool/runscript/runscript.py Updates XML emission to the new attribute names/structure (e.g., encoding_tag, optional template_options, instance encodings).
tests/result/test_result_parser.py Adjusts expectations for the new parsed structures/fields.
tests/result/test_result_classes.py Updates unit tests to match new dataclass signatures/fields (including keyword-only constructors).
tests/runscript/test_runscript_classes.py Updates XML-string assertions for renamed/added XML attributes and instance encodings.
tests/ref/test_eval.xml Updates the reference evaluation XML fixture to the new format.
docs/getting_started/workflow/index.md Rewords/moves an executable-permissions note into an admonition block.
docs/getting_started/gen/runscript.md Updates example encoding paths and clarifies path relativity.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/benchmarktool/runscript/runscript.py
Comment thread docs/getting_started/gen/runscript.md Outdated
Comment thread docs/getting_started/gen/runscript.md Outdated
tzschmidt and others added 3 commits May 22, 2026 14:31
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@tzschmidt tzschmidt requested review from BenKaufmann and rkaminsk May 22, 2026 12:56
Copy link
Copy Markdown
Contributor

@BenKaufmann BenKaufmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks ok to me, but let's wait until @rkaminsk had a look.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants