Skip to content

add infrastructre for executable examples, add results to the web page - #115

Draft
ev-br wants to merge 4 commits into
benchx-dev:mainfrom
ev-br:ci_exec_md
Draft

ev-br wants to merge 4 commits into
benchx-dev:mainfrom
ev-br:ci_exec_md

Conversation

@ev-br

@ev-br ev-br commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

This is a companion to benchx-dev/benchx_examples#16, which lays the ground work for making examples from benchx_examples executable.

The end goal is: that the examples stay current, and are executed on CI. Moreover:

  • benchx_examples CI runs examples for PRs in the _examples repo against the released benchx.
  • benchx PRs run examples in benchx CI
  • the doc build runs examples and places them to the docs website under a new "examples" heading.

To build the new doc site locally, use

$ python tools/build_site.py ../benchx_examples  
$ python -m http.server -d site 
# now open http://0.0.0.0:8000/

which expects to find the examples checkout of benchx-dev/benchx_examples#16 at ../benchx_examples.

Once the example PR lands, I'll change this one to look at benchx_examples main not a PR branch.

To keep the diffs small, I only hooked up a single example for now.

The locally built page looks like this:

Screenshot from 2026-11-14 19-55-22

Of course, examples themselves might need a rewrite for readability, which I also did not do just yet (to keep the diff sizes under control).

@ev-br
ev-br marked this pull request as draft November 14, 2026 17:23
@ev-br ev-br changed the title WIP: add reusable workflows for CI add infrastructre for executable examples, add results to the web page Nov 14, 2026
@ev-br
ev-br marked this pull request as ready for review November 14, 2026 20:18
Comment thread .github/workflows/e2e.yml
path: benchx_examples/examples/pyperf-suite/out
if-no-files-found: warn
retention-days: 14
uses: ev-br/benchx_examples/.github/workflows/reusable-example.yml@ci_exec_md

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocker until benchx-dev/benchx_examples#16 lands: with the new push: main trigger this runs a fork's mutable branch on main, and its output is what the site publishes.

fi
done <<< "$names"
done <<< "$runs"
if [ "$count" -eq 0 ]; then

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On main this deploys an Examples page that says "No example was built". That happens on the merge of this PR (docs build ~15s vs ~45s for pyperf-suite, and no earlier run on main has the artifact), after 14 quiet days, and on any gh failure hidden by || true. Fail, or skip the deploy, on main when nothing was found?

# it succeeded. The run for the push being built is usually still going, so the page
# is typically the one from the push before. Artifacts expire after 14 days; an example
# without an artifact is left out of the site, with a warning.
- name: Download the example pages from the latest e2e runs on main

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could build instead needs: a job that calls the reusable workflow, and download docs-* from the same run? That drops the cross-run lookup, actions: read, the one-push lag, the artifact expiry and the two-step manual redeploy, for ~1 min per docs build (more as examples are added).


- id: deployment
uses: actions/deploy-pages@v4
uses: actions/deploy-pages@368f82528645a54fb793d4d04e342629a3f51346 # v5.0.1

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The action major bumps are unrelated to the examples, and configure-pages / deploy-pages are not exercised here (deploy is skipped on PRs). Separate PR?

Comment thread tools/build_site.py
notes.append(f"{workflow} not found, so there is no list of the examples that CI runs "
"(name them with -e NAME)")
return []
names = re.findall(r"^\s+example:\s*([A-Za-z0-9_-]+)\s*$", workflow.read_text(), re.M)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This script is not run in CI and depends on the layout of another repo's e2e.yml, so it can break unnoticed. A CI smoke run (-e pyperf-suite) would cover it.

@ev-br

ev-br commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks for the review @rok !
As discussed earlier today, I'll look into simplifying this a bit:

  • move examples back to the benchx repository
  • simplify the workflows for a single-repo setup
  • add executable examples to the webpage, starting with the one here.

Other examples in benchx_examples need a work-through to make them runnable. Those which take longer will then stay in _examples in a cookbook-type state.

@ev-br
ev-br marked this pull request as draft November 19, 2026 14:14

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants