From a77dba1c9fa77143149e16e4df072cd16eb68a7d Mon Sep 17 00:00:00 2001 From: Rok Mihevc Date: Sat, 21 Nov 2026 15:14:32 +0100 Subject: [PATCH 1/3] DOC: add CI execution design doc and CI job driver role --- docs/design/README.md | 1 + docs/design/benchmark-environments.md | 1 + docs/design/ci-execution.md | 248 ++++++++++++++++++++++++++ docs/design/system-decomposition.md | 14 ++ zensical.toml | 1 + 5 files changed, 265 insertions(+) create mode 100644 docs/design/ci-execution.md diff --git a/docs/design/README.md b/docs/design/README.md index 8bded9e5..5c5b4bdc 100644 --- a/docs/design/README.md +++ b/docs/design/README.md @@ -12,6 +12,7 @@ designed around it (see the repository README for the philosophy). | [`harness-adapter.md`](harness-adapter.md) | Translating native harness output into results: the context document, mapping rules, driving half, context collectors, adapter catalog | draft | | [`benchmark-environments.md`](benchmark-environments.md) | Who prepares the environment, what the runner controls on its launched process, and what benchx records, with recommendations per kind of environment | draft | | [`comparator.md`](comparator.md) | Verdicts in history mode and run mode: the eligibility guard, dispersion-scaled verdicts, the comparison document | draft | +| [`ci-execution.md`](ci-execution.md) | Unattended CI benchmark execution: the provisioning handoff, job driver lifecycle, durable unit staging and restarts, trust boundaries | draft | | [`prototype-scope.md`](prototype-scope.md) | What the minimal implementation demonstrates and the decisions locked for it | agreed | | [`prototype-design.md`](prototype-design.md) | How the implementation is built: one CLI over a Python API, a Parquet store, three JSON documents | draft | | [`measurement-result-parquet.md`](measurement-result-parquet.md) | The Parquet form of a result: the mapping from the result schema to Arrow types, the message columns of the store's row | agreed | diff --git a/docs/design/benchmark-environments.md b/docs/design/benchmark-environments.md index 75026587..90886b34 100644 --- a/docs/design/benchmark-environments.md +++ b/docs/design/benchmark-environments.md @@ -170,6 +170,7 @@ A machine reserved for benchmarking, as in Apache Arrow's Conbench deployment, i Self-hosted runners provisioned on demand, as OpenBLAS does through cirun, are the practical way for a volunteer project to reach several architectures. They sit between hosted runners and dedicated nodes. - Fix the instance type and declare it. For an ephemeral machine, identity is the instance type and image rather than the host, since no host outlives a job. +- Assign a stable logical machine name before benchx runs (by setting the hostname or declaring the runner name) and prefer single-CPU-model instance families. Fresh cloud hostnames otherwise start a new series on every boot; see `ci-execution.md` §3 for the provisioning handoff. - Avoid burstable instance families. Their CPU allocation depends on a credit balance, which is invisible in the results. - Prefer instance types that map to whole sockets or bare metal, and dedicated tenancy where the budget allows. Neighbors on the same host are the dominant noise source and cannot be detected from inside the guest. - Expect each instance to be different hardware. Two instances of one type are two machines; series on such environments have a wider floor than on one physical node, and the comparator's noise estimate will reflect it. diff --git a/docs/design/ci-execution.md b/docs/design/ci-execution.md new file mode 100644 index 00000000..40ff60e3 --- /dev/null +++ b/docs/design/ci-execution.md @@ -0,0 +1,248 @@ +# CI Execution + +**Status:** Draft for review +**Companion to:** `system-decomposition.md` §3.4–§3.5, `runner.md`, `benchmark-environments.md` §5.2–§5.4, `comparator.md` +**Author:** Rok Mihevc + +## 1. Role + +The **CI job driver** is the unattended, non-interactive counterpart to the +workbench (`system-decomposition.md` §3.4). It carries out one remote benchmark +job on a CI-provisioned compute node and delivers its results durably, even +across crashes, timeouts, or preempted instances. + +Two boundaries govern this design (`system-decomposition.md` §5, `runner.md` +§7): + +- **CI platforms and machine provisioning are not components of benchx.** + Launching cloud VMs, registering CI runners, enforcing wall-clock caps, and + managing secrets belong to the CI platform and its cloud integration. +- **The runner remains unchanged.** It executes one resolved work order against + a prepared target on whatever machine it is given, and never builds, + provisions, or queues. + +What belongs in benchx's design is the contract where CI meets benchx: the +**provisioning handoff** (L0/L1 into benchx), the **job driver's lifecycle**, +**durable delivery and restart semantics**, and the **trust boundary**. + +## 2. Overview + +```text + Events (cron, PR, /bench) L0/L1 (CI platform + cloud) + │ │ + ▼ ▼ provisions & names node + ┌─────────────┐ ┌───────────────┐ + │ Scheduler │───triggers───▶│ CI job driver │ (requester: ci) + │ (or CI YAML)│ └───────┬───────┘ + └─────────────┘ │ 1. declare machine / L0–L1 facts + │ 2. call target provider (prepare) + │ 3. emit work orders (unit / round) + ▼ + ┌───────────────┐ raw staging ┌───────┐ + │ Runner + │───or ingest────▶│ Store │ + │ Adapters │ └───┬───┘ + └───────────────┘ │ + ▼ + Comparator / + Views / Alerts +``` + +Mapped onto the environment layers of `benchmark-environments.md` §2.1: + +| Layer | Responsibility in CI | Owner | +|---|---|---| +| **L0 Provisioning** | Launch and terminate VM/host, assign logical machine name, enforce max-lifetime watchdog, attach least-privilege role | CI platform / provisioner (outside benchx) | +| **L1 Machine session** | Apply OS/kernel/governor tuning (or use a fixed image), declare L0/L1 facts | Provisioner / image | +| **L2 Software environment** | Check out revision(s), set up dependencies, build with cache via target provider (`prepare`) | Target provider (`system-decomposition.md` §3.4) | +| **L3 Process** | Emit work orders (`requester.kind: "ci"`), enforce launch settings on measured processes | CI job driver → Runner | +| **L4–L5 Measurement** | Run harness per unit/round, capture context, emit schema results | Runner + Harness adapters | +| **Delivery & publishing** | Upload raw unit archives incrementally; compact/ingest into store; derive views and verdicts | CI job driver → Store ingest / Comparator | + +## 3. Provisioning handoff (L0/L1 to benchx) + +The job driver is agnostic to how the compute node is provisioned or entered. +Three common deployment shapes sit behind the same handoff boundary: + +| Shape | How a job reaches the node | Strengths | Tradeoffs | +|---|---|---|---| +| **Dedicated pull node** | A long-lived physical or tuned host polls a repo or queue (e.g. systemd timer) | Lowest noise floor (fixed physical host and L1 tuning); no CI runner agent or cloud token needed | Fixed hardware capacity; operator maintains OS, disk, and updates | +| **Ephemeral self-hosted CI runner** | A CI job launches a fresh cloud VM, registers a one-shot runner, executes the job, and terminates the VM | Pay-per-run; parallel bursts for backfill or PRs; clean state every run; native CI logs and UI | Higher host-to-host variance; requires a runner-registration token and leak-safe teardown | +| **Direct cloud job (no CI runner agent)** | A CI workflow invokes cloud APIs (e.g. `RunInstances` + user-data, SSM, or Batch) and lets the VM self-terminate | Pay-per-run and parallel; cleanest security (cloud OIDC only, no CI runner token); reliable VM self-termination | Worse CI UX unless logs/status are streamed or polled from cloud storage | + +Persistent, always-on self-hosted CI runners on cloud VMs are generally a poor +fit for public repositories: they pay 24/7 for idle capacity, accumulate state +drift, carry security risks on pull requests, and lose their same-host noise +advantage whenever stopped and started on a new physical host. + +Whichever shape is used, the provisioner must satisfy three handoff rules +before invoking the job driver: + +1. **Stable logical machine identity.** Ephemeral cloud instances receive a + fresh hostname on every boot, which would split every run into a new series. + The provisioner must assign a stable logical machine name bound to one + instance type / hardware class (by setting the hostname or declaring the + runner name) and prefer single-CPU-model instance families + (`benchmark-environments.md` §5.4). +2. **Declared L0/L1 facts.** Facts the runner cannot detect — instance type, + tenancy, base image or AMI identifier, and tuning playbook version — are + written to the node's declared-facts file (e.g. `.benchx/setup.json`) or + passed into the work order before execution starts. +3. **Leak-safe lifecycle.** For ephemeral instances, provisioning and teardown + bracket the job (`start` and an `always` `stop` step), backed by an + instance-side max-lifetime shutdown/self-termination guard and periodic + cleanup of orphaned instances so a failed CI workflow cannot leak compute. + +## 4. Job driver lifecycle + +A CI job runs in one of two modes, matching the comparator's two modes +(`comparator.md` §1): + +### 4.1 Tracking mode (continuous history) + +Measures one revision on one logical machine to extend per-series history: + +1. **Resolve inputs:** pin the target commit SHA, logical machine name, suite + filter, and wall-clock budget. +2. **Inspect prior progress (for restarts):** list already-recorded units for + `(commit, machine)` so completed work is skipped (§5). +3. **Prepare target:** invoke the project's target provider (`prepare`) for the + pinned commit, using any configured build cache (e.g. `sccache` in object + storage). Record resolved software-environment facts (such as a lockfile or + explicit package list) alongside L0/L1 declarations. +4. **Execute units:** for each remaining unit (such as one benchmark binary or + one suite command) while within the wall-clock budget, author a resolved + work order (`requester: {"kind": "ci", "name": "..."}`), run it through the + runner (`--no-ingest` when staging raw outputs), and upload the unit's + results immediately. +5. **Record attempt outcome:** write an attempt summary and log (including any + provider build failure, unit crash, or budget cutoff) and trigger store + compaction/publishing. + +### 4.2 Comparison mode (PR or A/B verification) + +Evaluates a contender revision against a baseline on the same machine within +one job (`comparator.md` §2.3, UC-03): + +1. **Prepare both targets** on the same compute node through the target + provider (`prepare` for baseline and contender). +2. **Run an interleaved session:** share one `run_key`, emit one work order per + side and round (`round`, `slot`, `requester.kind: "ci"`), and alternate + sides across rounds — the same session loop the workbench uses + (`system-decomposition.md` §3.4). +3. **Ingest and compare:** ingest the run's results into an ephemeral or + project store, invoke the comparator in run mode, and hand the resulting + comparison document to alerting (e.g. a PR comment). + +### 4.3 Build failures vs. benchmark failures + +Following `system-decomposition.md` §3.4 and `runner.md` §6: + +- A **target-provider failure** (checkout, dependency setup, or compile error) + happens before any work order exists. The job driver records it in the + attempt metadata and log and exits; it does not synthesize runner results. +- A **benchmark case failure or timeout** inside a unit is handled by the + runner and recorded as an explicit schema result (`status: error` or + `timeout`). A unit that completed with some failing cases is a finished unit + and is not retried. +- A **unit crash** (binary failed to start or produced no results) is recorded + as a failed unit attempt so restarts know whether to retry up to the attempt + cap. + +## 5. Durable delivery, partial runs, and restarts + +Long benchmark suites on ephemeral machines will hit job timeouts, spot +preemptions, or hanging binaries. Delivery is designed so completed work is +never lost and restarts are safe: + +1. **Unit-level incremental staging.** Each unit's raw result documents and + work orders are uploaded to durable staging (such as object storage) as soon + as the unit finishes, rather than buffered until the entire job completes. + At most the single unit in flight is lost on crash or timeout. +2. **Append-only, write-once staging.** The compute node only adds objects: + write-once unit archives (enforced by storage policy where available, e.g. + conditional put) plus per-attempt metadata and logs. It never reads back, + merges, or mutates the compacted store. +3. **Forward-only restarts (the ratchet).** Run status is derived from the + staged objects for `(commit, machine)` — no separate queue database is + required. Re-dispatching the same pair skips every unit that already has a + staged archive, paying for setup and a cached build again but never + re-measuring finished units. Concurrency control per `(commit, machine)` + prevents concurrent duplicate attempts, with write-once storage as the + backstop. +4. **Idempotent compaction and ingest.** A separate, serialized job (typically + on a hosted CI runner) reads staged unit archives, runs `bx ingest` into the + result store (e.g. one Parquet file per `(commit, machine)` or a store + service), and updates derived views and alerts. Because store ingest is + idempotent (`benchmark-result-schema.md` §5.4), partial runs can be + published immediately and re-compacted safely whenever later attempts finish. + +For small, fast suites where incremental staging is unnecessary, a CI job may +alternatively ingest directly into a local or remote store at job end; the +contract with the runner and store is identical. + +## 6. Trust and security boundary + +Benchmarking executes arbitrary code from the repository under test. Any CI +deployment that accepts pull requests or runs on a public repository must +enforce three boundaries: + +1. **Trigger gating.** Automated tracking runs follow trusted branches (e.g. + `main`). Pull-request benchmark runs must be gated by explicit authorization + (for example, a comment command restricted to maintainers or collaborators), + never triggered automatically from unreviewed fork PRs. +2. **Least-privilege credential split.** + - *CI control plane / workflow role:* may launch and terminate instances + scoped to the project's tags/network, read staging, and write to the + compacted store and published views. + - *Compute node role:* may only append to the raw staging prefix (and read/write + a build cache such as `sccache`). It must **not** hold credentials to + modify or delete compacted store history, push to git repositories, or + administer CI runners. +3. **Secret hygiene.** The runner records only allowlisted environment + variables (`benchmark-environments.md` §3.1) so CI tokens are never swept + into results. Provisioning scripts and target providers must never place + secrets in declared provenance (`how_built`, `activation`, or + `.benchx/setup.json`). + +## 7. Reference deployment mappings + +The same contract supports minimal dedicated deployments and cloud-ephemeral +deployments without changing benchx itself: + +| Abstract role | Ephemeral GitHub Actions + EC2 + S3 | Dedicated systemd node + Git | +|---|---|---| +| **Scheduler / trigger** | Scheduled & comment-dispatch workflows checking staged objects | Cron / systemd timer + file queue in repo | +| **Provisioner (L0/L1)** | `start`/`stop` jobs (ephemeral self-hosted runner or direct EC2 launch) + hostname & max-lifetime watchdog | Fixed physical host, tuned once by operator | +| **Target provider (L2)** | Commit-specific env setup + incremental build with S3 `sccache` | Reused worktree + local incremental build | +| **CI job driver** | Unit loop with wall-clock budget, `bx bench --no-ingest`, write-once S3 upload | Per-commit script calling `bx bench` | +| **Compaction / store** | Serialized hosted publish workflow running `bx ingest` into Parquet on S3 | Local `bx ingest` into Parquet committed to repo | +| **Views & alerts** | Derived tables + static site / PR comments | Derived tables + static site | + +## 8. Boundaries + +- **Not the scheduler:** does not decide which commits to benchmark, manage + fleet priorities, or perform regression bisection (`system-decomposition.md` + §3.5). +- **Not the runner:** does not execute harnesses or capture machine snapshots + itself; it authors work orders and invokes the runner (`runner.md`). +- **Not the target provider:** does not check out or build software itself; it + calls the project's target provider before issuing work orders. +- **Not the provisioner:** does not prescribe a specific CI platform or cloud + provider; it defines the handoff contract any provisioner can satisfy. +- **Not the store:** stages raw results and triggers ingest; identity + assignment, validation, and deduplication remain in the store. + +## 9. Open questions + +1. **Reusable driver vs. deployment scripts:** Should benchx ship a thin + reusable CLI helper for the unit-budget loop and attempt manifest, or should + the CI job driver stay a small deployment-side script calling `bx bench`, + `bx run`, and `bx ingest`? +2. **Attempt and staging manifest schema:** Should the staging layout and + attempt record (units expected, units completed, provider failure log, + status derivation) become a versioned benchx schema, or remain a convention + of object-storage deployments? +3. **Cross-revision environments in PR comparison mode:** When baseline and + contender revisions require different dependency sets on an ephemeral + instance, how should the job driver budget setup/build time against + measurement rounds? diff --git a/docs/design/system-decomposition.md b/docs/design/system-decomposition.md index 876deb27..b8aaa68a 100644 --- a/docs/design/system-decomposition.md +++ b/docs/design/system-decomposition.md @@ -35,6 +35,10 @@ Two rules shape the decomposition: ┌────────────┐ ▲ └───────┘ │ Importers │────────┼──────────────▲ └────────────┘ │ + ┌───────────────┐ + │ CI job driver │ runs one unattended job on a CI node + └───────────────┘ + ▲ ┌───────────┐ │ Scheduler │ decides what runs, where, when └───────────┘ @@ -232,6 +236,16 @@ job, or script may author work orders directly. The scheduler exists for contention — shared fleets, queues, priorities, automated backfill — and nothing else in the system depends on its existence. +**CI job driver (unattended counterpart to the workbench).** Between a +scheduler or CI trigger and the runner on a remote node sits a thin job driver +(`ci-execution.md`) using `requester.kind: "ci"`. Like the workbench locally, +it calls the target provider (`prepare`), authors resolved work orders, and +invokes the runner; for unattended and ephemeral CI machines it also handles +the provisioning handoff (stable logical machine identity and declared L0/L1 +facts), time-budgeted and resumable unit execution, and incremental staging for +idempotent store ingest. CI platforms and machine provisioning themselves +remain outside benchx (§5). + ### 3.6 Comparator **Role:** Answer "did performance change between A and B?" defensibly. diff --git a/zensical.toml b/zensical.toml index 28697005..77c17592 100644 --- a/zensical.toml +++ b/zensical.toml @@ -37,6 +37,7 @@ nav = [ { "Harness adapter" = "design/harness-adapter.md" }, { "Benchmark environments" = "design/benchmark-environments.md" }, { "Comparator" = "design/comparator.md" }, + { "CI execution" = "design/ci-execution.md" }, { "Prototype scope" = "design/prototype-scope.md" }, { "Prototype design" = "design/prototype-design.md" }, { "Result in Parquet" = "design/measurement-result-parquet.md" }, From 9fa8a4e156b7e7e5b535c8a60411681b50a342b8 Mon Sep 17 00:00:00 2001 From: Rok Mihevc Date: Sat, 21 Nov 2026 15:21:32 +0100 Subject: [PATCH 2/3] DOC: clean up overview diagram in system-decomposition.md --- docs/design/system-decomposition.md | 28 +++++++++++++--------------- 1 file changed, 13 insertions(+), 15 deletions(-) diff --git a/docs/design/system-decomposition.md b/docs/design/system-decomposition.md index b8aaa68a..9c8bc2b6 100644 --- a/docs/design/system-decomposition.md +++ b/docs/design/system-decomposition.md @@ -27,21 +27,19 @@ Two rules shape the decomposition: ## 2. Overview ``` - produce results consume results - ┌────────────┐ ┌────────────┐ ┌───────┐ ┌────────────┐ ┌──────────┐ - │ Workbench │ │ Runner + │ │ │ │ Comparator │ │ Views │ - │ (local CLI)│─▶│ Adapters │──▶│ Store │──▶│ Detector │─▶│ Alerts │ - └────────────┘ └────────────┘ │ │ └────────────┘ └──────────┘ - ┌────────────┐ ▲ └───────┘ - │ Importers │────────┼──────────────▲ - └────────────┘ │ - ┌───────────────┐ - │ CI job driver │ runs one unattended job on a CI node - └───────────────┘ - ▲ - ┌───────────┐ - │ Scheduler │ decides what runs, where, when - └───────────┘ + produce results consume results + ┌───────────────┐ ┌────────────┐ ┌───────┐ ┌────────────┐ ┌──────────┐ + │ Workbench │─▶│ │ │ │ │ Comparator │ │ Views │ + │ (local CLI) │ │ Runner + │──▶│ Store │──▶│ Detector │─▶│ Alerts │ + └───────────────┘ │ Adapters │ │ │ └────────────┘ └──────────┘ + ┌───────────────┐ │ │ └───▲───┘ + │ CI job driver │─▶│ │ │ + │ (unattended) │ └────────────┘ │ + └───────▲───────┘ ┌───────────┐ │ + │ │ Importers │────────┘ + ┌───────┴───────┐ └───────────┘ + │ Scheduler │ decides what runs, where, when + └───────────────┘ ``` Everything to the left of the store produces schema results; everything to the From bab2e16d6001185434e069b16a554de6113d4f0e Mon Sep 17 00:00:00 2001 From: Rok Mihevc Date: Sat, 21 Nov 2026 15:36:22 +0100 Subject: [PATCH 3/3] DOC: frame CI execution as a CI workbench --- docs/design/README.md | 4 +- docs/design/benchmark-environments.md | 2 +- .../{ci-execution.md => ci-workbench.md} | 104 ++++++++++++------ docs/design/system-decomposition.md | 65 ++++++----- zensical.toml | 2 +- 5 files changed, 109 insertions(+), 68 deletions(-) rename docs/design/{ci-execution.md => ci-workbench.md} (75%) diff --git a/docs/design/README.md b/docs/design/README.md index 5c5b4bdc..4b445ad0 100644 --- a/docs/design/README.md +++ b/docs/design/README.md @@ -12,12 +12,12 @@ designed around it (see the repository README for the philosophy). | [`harness-adapter.md`](harness-adapter.md) | Translating native harness output into results: the context document, mapping rules, driving half, context collectors, adapter catalog | draft | | [`benchmark-environments.md`](benchmark-environments.md) | Who prepares the environment, what the runner controls on its launched process, and what benchx records, with recommendations per kind of environment | draft | | [`comparator.md`](comparator.md) | Verdicts in history mode and run mode: the eligibility guard, dispersion-scaled verdicts, the comparison document | draft | -| [`ci-execution.md`](ci-execution.md) | Unattended CI benchmark execution: the provisioning handoff, job driver lifecycle, durable unit staging and restarts, trust boundaries | draft | +| [`ci-workbench.md`](ci-workbench.md) | Unattended CI workbenches: the provisioning handoff, lifecycle, durable unit staging and restarts, trust and repository boundaries | draft | | [`prototype-scope.md`](prototype-scope.md) | What the minimal implementation demonstrates and the decisions locked for it | agreed | | [`prototype-design.md`](prototype-design.md) | How the implementation is built: one CLI over a Python API, a Parquet store, three JSON documents | draft | | [`measurement-result-parquet.md`](measurement-result-parquet.md) | The Parquet form of a result: the mapping from the result schema to Arrow types, the message columns of the store's row | agreed | Not yet designed (roles fixed in the decomposition, documents pending): -workbench, scheduler internals, change detector, alerting & annotations, +local workbench, scheduler internals, change detector, alerting & annotations, views, series continuity and identity-policy authoring, the environment plugin interface, importers (beyond schema §6). diff --git a/docs/design/benchmark-environments.md b/docs/design/benchmark-environments.md index 90886b34..93cf892d 100644 --- a/docs/design/benchmark-environments.md +++ b/docs/design/benchmark-environments.md @@ -170,7 +170,7 @@ A machine reserved for benchmarking, as in Apache Arrow's Conbench deployment, i Self-hosted runners provisioned on demand, as OpenBLAS does through cirun, are the practical way for a volunteer project to reach several architectures. They sit between hosted runners and dedicated nodes. - Fix the instance type and declare it. For an ephemeral machine, identity is the instance type and image rather than the host, since no host outlives a job. -- Assign a stable logical machine name before benchx runs (by setting the hostname or declaring the runner name) and prefer single-CPU-model instance families. Fresh cloud hostnames otherwise start a new series on every boot; see `ci-execution.md` §3 for the provisioning handoff. +- Assign a stable logical machine name before benchx runs (by setting the hostname or declaring the runner name) and prefer single-CPU-model instance families. Fresh cloud hostnames otherwise start a new series on every boot; see `ci-workbench.md` §3 for the provisioning handoff. - Avoid burstable instance families. Their CPU allocation depends on a credit balance, which is invisible in the results. - Prefer instance types that map to whole sockets or bare metal, and dedicated tenancy where the budget allows. Neighbors on the same host are the dominant noise source and cannot be detected from inside the guest. - Expect each instance to be different hardware. Two instances of one type are two machines; series on such environments have a wider floor than on one physical node, and the comparator's noise estimate will reflect it. diff --git a/docs/design/ci-execution.md b/docs/design/ci-workbench.md similarity index 75% rename from docs/design/ci-execution.md rename to docs/design/ci-workbench.md index 40ff60e3..1bebd607 100644 --- a/docs/design/ci-execution.md +++ b/docs/design/ci-workbench.md @@ -1,4 +1,4 @@ -# CI Execution +# CI Workbench **Status:** Draft for review **Companion to:** `system-decomposition.md` §3.4–§3.5, `runner.md`, `benchmark-environments.md` §5.2–§5.4, `comparator.md` @@ -6,24 +6,39 @@ ## 1. Role -The **CI job driver** is the unattended, non-interactive counterpart to the -workbench (`system-decomposition.md` §3.4). It carries out one remote benchmark -job on a CI-provisioned compute node and delivers its results durably, even -across crashes, timeouts, or preempted instances. +A **CI workbench** is an unattended workbench (`system-decomposition.md` §3.4) +that carries out one benchmark job on a CI-provisioned compute node and +delivers its results durably, even across crashes, timeouts, or preempted +instances. + +It sits in the same architectural slot as a local workbench (`bx` or +`spin bench`), wrapping the target provider, runner, store, and comparator for +an unattended remote environment instead of an interactive terminal: + +| Responsibility | Local workbench (`bx`, `spin`) | CI workbench (e.g. GitHub Actions + EC2) | +|---|---|---| +| **Trigger** | Developer at a terminal | Scheduler or CI event (cron, PR command, manual dispatch) | +| **Target prep** | Calls target provider (`prepare` / `describe`) | Calls target provider (`prepare`, typically with shared build cache) | +| **Loop above runner** | Single run or interactive comparison session | Time-budgeted, resumable unit loop (or PR comparison rounds) | +| **Work orders** | `requester.kind: "workbench"` | `requester.kind: "ci"` | +| **Delivery** | Local store (`store.parquet`) | Write-once raw staging (e.g. S3) → serialized idempotent compaction (`bx ingest`) | Two boundaries govern this design (`system-decomposition.md` §5, `runner.md` §7): -- **CI platforms and machine provisioning are not components of benchx.** - Launching cloud VMs, registering CI runners, enforcing wall-clock caps, and - managing secrets belong to the CI platform and its cloud integration. -- **The runner remains unchanged.** It executes one resolved work order against - a prepared target on whatever machine it is given, and never builds, - provisions, or queues. +- **It is a workbench, not a new kind of runner.** The runner (`runner.md`) is + already complete for remote and cloud nodes: it takes one resolved work order + against a prebuilt target on the current host and emits schema results. There + is no "EC2 runner" inside benchx — the same runner executes unchanged inside + an EC2 instance or a dedicated host. +- **CI platforms and cloud provisioning remain outside benchx.** Launching VMs, + registering CI runners, enforcing wall-clock caps, and managing cloud secrets + belong to the CI platform and its cloud integration. What belongs in benchx's design is the contract where CI meets benchx: the -**provisioning handoff** (L0/L1 into benchx), the **job driver's lifecycle**, -**durable delivery and restart semantics**, and the **trust boundary**. +**provisioning handoff** (L0/L1 into benchx), the **CI workbench lifecycle**, +**durable delivery and restart semantics**, the **trust boundary**, and **where +implementations live**. ## 2. Overview @@ -32,7 +47,7 @@ What belongs in benchx's design is the contract where CI meets benchx: the │ │ ▼ ▼ provisions & names node ┌─────────────┐ ┌───────────────┐ - │ Scheduler │───triggers───▶│ CI job driver │ (requester: ci) + │ Scheduler │───triggers───▶│ CI workbench │ (requester: ci) │ (or CI YAML)│ └───────┬───────┘ └─────────────┘ │ 1. declare machine / L0–L1 facts │ 2. call target provider (prepare) @@ -54,13 +69,13 @@ Mapped onto the environment layers of `benchmark-environments.md` §2.1: | **L0 Provisioning** | Launch and terminate VM/host, assign logical machine name, enforce max-lifetime watchdog, attach least-privilege role | CI platform / provisioner (outside benchx) | | **L1 Machine session** | Apply OS/kernel/governor tuning (or use a fixed image), declare L0/L1 facts | Provisioner / image | | **L2 Software environment** | Check out revision(s), set up dependencies, build with cache via target provider (`prepare`) | Target provider (`system-decomposition.md` §3.4) | -| **L3 Process** | Emit work orders (`requester.kind: "ci"`), enforce launch settings on measured processes | CI job driver → Runner | +| **L3 Process** | Emit work orders (`requester.kind: "ci"`), enforce launch settings on measured processes | CI workbench → Runner | | **L4–L5 Measurement** | Run harness per unit/round, capture context, emit schema results | Runner + Harness adapters | -| **Delivery & publishing** | Upload raw unit archives incrementally; compact/ingest into store; derive views and verdicts | CI job driver → Store ingest / Comparator | +| **Delivery & publishing** | Upload raw unit archives incrementally; compact/ingest into store; derive views and verdicts | CI workbench → Store ingest / Comparator | ## 3. Provisioning handoff (L0/L1 to benchx) -The job driver is agnostic to how the compute node is provisioned or entered. +A CI workbench is agnostic to how the compute node is provisioned or entered. Three common deployment shapes sit behind the same handoff boundary: | Shape | How a job reaches the node | Strengths | Tradeoffs | @@ -75,7 +90,7 @@ drift, carry security risks on pull requests, and lose their same-host noise advantage whenever stopped and started on a new physical host. Whichever shape is used, the provisioner must satisfy three handoff rules -before invoking the job driver: +before invoking the CI workbench: 1. **Stable logical machine identity.** Ephemeral cloud instances receive a fresh hostname on every boot, which would split every run into a new series. @@ -92,9 +107,9 @@ before invoking the job driver: instance-side max-lifetime shutdown/self-termination guard and periodic cleanup of orphaned instances so a failed CI workflow cannot leak compute. -## 4. Job driver lifecycle +## 4. CI workbench lifecycle -A CI job runs in one of two modes, matching the comparator's two modes +A CI workbench runs in one of two modes, matching the comparator's two modes (`comparator.md` §1): ### 4.1 Tracking mode (continuous history) @@ -127,7 +142,7 @@ one job (`comparator.md` §2.3, UC-03): provider (`prepare` for baseline and contender). 2. **Run an interleaved session:** share one `run_key`, emit one work order per side and round (`round`, `slot`, `requester.kind: "ci"`), and alternate - sides across rounds — the same session loop the workbench uses + sides across rounds — the same session loop a local workbench uses (`system-decomposition.md` §3.4). 3. **Ingest and compare:** ingest the run's results into an ephemeral or project store, invoke the comparator in run mode, and hand the resulting @@ -138,7 +153,7 @@ one job (`comparator.md` §2.3, UC-03): Following `system-decomposition.md` §3.4 and `runner.md` §6: - A **target-provider failure** (checkout, dependency setup, or compile error) - happens before any work order exists. The job driver records it in the + happens before any work order exists. The CI workbench records it in the attempt metadata and log and exits; it does not synthesize runner results. - A **benchmark case failure or timeout** inside a unit is handled by the runner and recorded as an explicit schema result (`status: error` or @@ -176,8 +191,8 @@ never lost and restarts are safe: idempotent (`benchmark-result-schema.md` §5.4), partial runs can be published immediately and re-compacted safely whenever later attempts finish. -For small, fast suites where incremental staging is unnecessary, a CI job may -alternatively ingest directly into a local or remote store at job end; the +For small, fast suites where incremental staging is unnecessary, a CI workbench +may alternatively ingest directly into a local or remote store at job end; the contract with the runner and store is identical. ## 6. Trust and security boundary @@ -194,17 +209,34 @@ enforce three boundaries: - *CI control plane / workflow role:* may launch and terminate instances scoped to the project's tags/network, read staging, and write to the compacted store and published views. - - *Compute node role:* may only append to the raw staging prefix (and read/write - a build cache such as `sccache`). It must **not** hold credentials to - modify or delete compacted store history, push to git repositories, or - administer CI runners. + - *Compute node role:* may only append to the raw staging prefix (and + read/write a build cache such as `sccache`). It must **not** hold + credentials to modify or delete compacted store history, push to git + repositories, or administer CI runners. 3. **Secret hygiene.** The runner records only allowlisted environment variables (`benchmark-environments.md` §3.1) so CI tokens are never swept into results. Provisioning scripts and target providers must never place secrets in declared provenance (`how_built`, `activation`, or `.benchx/setup.json`). -## 7. Reference deployment mappings +## 7. Packaging and repository boundaries + +To keep the core `benchx` package small, cloud-free, and CI-vendor-neutral, +responsibilities split across three layers: + +1. **Core `benchx` repository:** schemas, runner, adapters, local workbench + (`bx`), store, comparator, and this design contract. Contains no cloud + templates (CloudFormation/Terraform) or CI-platform workflows. +2. **Reusable CI workbench repository (companion repo, once shared across + projects):** generic CI workflows, infrastructure templates (e.g. AWS + setup), unit-budget ratchet, and staging compaction scripts parameterized by + project configuration. +3. **Project benchmark repository (e.g. per-project tracking repo):** + project-specific files only — `benchx.conf.json`, the project's target + provider (`provider/`), machine catalog (`machines.json`), and project + metadata. + +## 8. Reference deployment mappings The same contract supports minimal dedicated deployments and cloud-ephemeral deployments without changing benchx itself: @@ -214,11 +246,11 @@ deployments without changing benchx itself: | **Scheduler / trigger** | Scheduled & comment-dispatch workflows checking staged objects | Cron / systemd timer + file queue in repo | | **Provisioner (L0/L1)** | `start`/`stop` jobs (ephemeral self-hosted runner or direct EC2 launch) + hostname & max-lifetime watchdog | Fixed physical host, tuned once by operator | | **Target provider (L2)** | Commit-specific env setup + incremental build with S3 `sccache` | Reused worktree + local incremental build | -| **CI job driver** | Unit loop with wall-clock budget, `bx bench --no-ingest`, write-once S3 upload | Per-commit script calling `bx bench` | +| **CI workbench** | Unit loop with wall-clock budget, `bx bench --no-ingest`, write-once S3 upload | Per-commit script calling `bx bench` | | **Compaction / store** | Serialized hosted publish workflow running `bx ingest` into Parquet on S3 | Local `bx ingest` into Parquet committed to repo | | **Views & alerts** | Derived tables + static site / PR comments | Derived tables + static site | -## 8. Boundaries +## 9. Boundaries - **Not the scheduler:** does not decide which commits to benchmark, manage fleet priorities, or perform regression bisection (`system-decomposition.md` @@ -232,11 +264,11 @@ deployments without changing benchx itself: - **Not the store:** stages raw results and triggers ingest; identity assignment, validation, and deduplication remain in the store. -## 9. Open questions +## 10. Open questions -1. **Reusable driver vs. deployment scripts:** Should benchx ship a thin - reusable CLI helper for the unit-budget loop and attempt manifest, or should - the CI job driver stay a small deployment-side script calling `bx bench`, +1. **Reusable CLI helpers vs. deployment scripts:** Should core `benchx` ship a + thin cloud-agnostic helper for the unit-budget loop and attempt manifest, or + should CI workbenches stay small deployment-side scripts calling `bx bench`, `bx run`, and `bx ingest`? 2. **Attempt and staging manifest schema:** Should the staging layout and attempt record (units expected, units completed, provider failure log, @@ -244,5 +276,5 @@ deployments without changing benchx itself: of object-storage deployments? 3. **Cross-revision environments in PR comparison mode:** When baseline and contender revisions require different dependency sets on an ephemeral - instance, how should the job driver budget setup/build time against + instance, how should a CI workbench budget setup/build time against measurement rounds? diff --git a/docs/design/system-decomposition.md b/docs/design/system-decomposition.md index 9c8bc2b6..4db11283 100644 --- a/docs/design/system-decomposition.md +++ b/docs/design/system-decomposition.md @@ -27,19 +27,19 @@ Two rules shape the decomposition: ## 2. Overview ``` - produce results consume results - ┌───────────────┐ ┌────────────┐ ┌───────┐ ┌────────────┐ ┌──────────┐ - │ Workbench │─▶│ │ │ │ │ Comparator │ │ Views │ - │ (local CLI) │ │ Runner + │──▶│ Store │──▶│ Detector │─▶│ Alerts │ - └───────────────┘ │ Adapters │ │ │ └────────────┘ └──────────┘ - ┌───────────────┐ │ │ └───▲───┘ - │ CI job driver │─▶│ │ │ - │ (unattended) │ └────────────┘ │ - └───────▲───────┘ ┌───────────┐ │ - │ │ Importers │────────┘ - ┌───────┴───────┐ └───────────┘ - │ Scheduler │ decides what runs, where, when - └───────────────┘ + produce results consume results + ┌─────────────────┐ ┌────────────┐ ┌───────┐ ┌────────────┐ ┌──────────┐ + │ Local workbench │─▶│ │ │ │ │ Comparator │ │ Views │ + │ (local CLI) │ │ Runner + │──▶│ Store │──▶│ Detector │─▶│ Alerts │ + └─────────────────┘ │ Adapters │ │ │ └────────────┘ └──────────┘ + ┌─────────────────┐ │ │ └───▲───┘ + │ CI workbench │─▶│ │ │ + │ (unattended) │ └────────────┘ │ + └────────▲────────┘ ┌───────────┐ │ + │ │ Importers │────────┘ + ┌────────┴────────┐ └───────────┘ + │ Scheduler │ decides what runs, where, when + └─────────────────┘ ``` Everything to the left of the store produces schema results; everything to the @@ -109,7 +109,12 @@ harness did not report. ### 3.4 Workbench -**Role:** The developer's local interface for one-off benchmarking. +**Role:** The orchestrator above the runner and comparator for a given +execution context — either interactive local use on a developer's machine +(**local workbench**) or unattended execution on a CI-provisioned node +(**CI workbench**). + +A **local workbench** (such as `bx` or a project front end like `spin bench`): - Runs and compares benchmarks on the developer's own machine, entirely offline: current workspace against a baseline build, one build configuration @@ -135,10 +140,24 @@ harness did not report. - Can promote a kept local result into the result store unchanged — the local file and the stored result are the same schema object. -**Does not:** require a server, an account, or any part of the fleet, or -build anything itself. -It is the runner plus comparator wrapped for human, interactive use, plus the -session loop and the call to a target provider. +A **CI workbench** (`ci-workbench.md`) plays the same orchestrating role for +unattended jobs triggered by a scheduler or CI workflow (`requester.kind: +"ci"`): + +- Calls the target provider (`prepare`), authors resolved work orders, and + invokes the same runner unchanged on the CI-provisioned node (there is no + separate "EC2 runner" or CI-specific runner). +- Handles what unattended and ephemeral machines require: the provisioning + handoff (stable logical machine identity and declared L0/L1 facts), + time-budgeted and resumable unit execution (or interleaved PR comparison + rounds), and incremental raw staging for idempotent store ingest. +- Leaves CI triggering/queuing to the scheduler (§3.5) and cloud VM + provisioning to the CI platform (§5). + +**Does not:** build anything itself or provision cloud machines. A local +workbench requires no server, account, or fleet. A workbench is the runner plus +comparator wrapped for a specific workflow, plus its execution loop and calls +to a target provider. **Target provider (extension point, not a component).** The runner never builds, so something must turn a source ref such as `main` into a prepared @@ -234,16 +253,6 @@ job, or script may author work orders directly. The scheduler exists for contention — shared fleets, queues, priorities, automated backfill — and nothing else in the system depends on its existence. -**CI job driver (unattended counterpart to the workbench).** Between a -scheduler or CI trigger and the runner on a remote node sits a thin job driver -(`ci-execution.md`) using `requester.kind: "ci"`. Like the workbench locally, -it calls the target provider (`prepare`), authors resolved work orders, and -invokes the runner; for unattended and ephemeral CI machines it also handles -the provisioning handoff (stable logical machine identity and declared L0/L1 -facts), time-budgeted and resumable unit execution, and incremental staging for -idempotent store ingest. CI platforms and machine provisioning themselves -remain outside benchx (§5). - ### 3.6 Comparator **Role:** Answer "did performance change between A and B?" defensibly. diff --git a/zensical.toml b/zensical.toml index 77c17592..75a23e43 100644 --- a/zensical.toml +++ b/zensical.toml @@ -37,7 +37,7 @@ nav = [ { "Harness adapter" = "design/harness-adapter.md" }, { "Benchmark environments" = "design/benchmark-environments.md" }, { "Comparator" = "design/comparator.md" }, - { "CI execution" = "design/ci-execution.md" }, + { "CI workbench" = "design/ci-workbench.md" }, { "Prototype scope" = "design/prototype-scope.md" }, { "Prototype design" = "design/prototype-design.md" }, { "Result in Parquet" = "design/measurement-result-parquet.md" },