Skip to content

Add logging - #127

Open
ev-br wants to merge 12 commits into
benchx-dev:mainfrom
ev-br:logging-phases-1-2
Open

ev-br wants to merge 12 commits into
benchx-dev:mainfrom
ev-br:logging-phases-1-2

Conversation

@ev-br

@ev-br ev-br commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator

closes gh-85

Add basic logging to benchx. The goal is to make it easier to trace what benchx does under the hood, and make it easier to control the verbosity of the terminal output (cf #117) without loosing the ability to trace the details (what benchx is actually doing, where do various parameters come from etc etc).

Two examples of where it's helpful:

All output is routed through the logger. There are two handlers: file and console. By default, console gets messages INFO level and above, the file in out/ gets DEBUG level and above.

Logging can be controlled via CLI or benchx.conf.json or environment variables, with a usual precedence:

Control Setting
--log-level LEVEL debug (default), info, warning or error, in any case
$BENCHX_LOG_LEVEL Same values, used when no flag is given
log_level in benchx.conf.json Same values, used when neither of the above is given

The order is flag, then environment, then file, then debug. A bad environment value is refused (BENCHX_LOG_LEVEL=loud: not one of …). --no-log --log-level … is a usage error, since there is nothing to set a level on.

What each level keeps
  • ERROR: refusals, crashes with their traceback, and a store write that fails.
  • WARNING: failed cases and cases that exit non-zero, rejected documents, failed invariants, excluded results,
    fallbacks, and the footer's "left undone" phrases (1 of 4 cases not measured).
  • INFO: the start and end records (the end carries the exit code), the config line, the run key, each case's start and
    result line, the order summary, and each unit's verdict.
  • DEBUG: everything else: settings with their sources, parsed arguments, environment variables consulted, command
    lines, workload parameters, file paths, harness output, git commands, and store reads.
  • --log-level warning drops the start and end records, because they are INFO.
Where the log goes
  • Default: {out}/bx.log in the run's output folder, for bench, compare and run --out. Other commands log
    nothing unless asked.
  • Switch: --no-log turns it off, --log sends it to stderr, --log-file PATH appends to a file ({out} and
    {root} work in the path). The config key logging takes false, true or a path. Flags beat the file.
  • Pre-start records: records made before the log starts are held in a buffer and written into it once it does.

CLI verbosity controls

Additionally, bx bench and bx compare get new CLI arguments, -v / -q to control verbosity of the terminal output.
These options only change what bx writes to stderr.

Option Terminal shows
(none) What bx printed before logging existed: progress, refusals, rejections
-q Warnings and errors only; INFO progress is hidden
-qq Errors only
-v Every INFO record, as a bare message: case lines, run key, config, verdicts
-vv Every DEBUG record too
-vvv, -vvvv, … Same as -vv, and each extra v also asks the harness to say more: -vvv is --harness-verbosity 1, -vvvv is --harness-verbosity 2
  • Net count: -v and -q add up, so -v -q is the default. $BENCHX_VERBOSE sets the net count (-2 or more) when no
    flag is given.
  • The log file: -v and -q never change it. Its content is set by --log-level (default debug).
  • --log: -v and -q are ignored, because stderr is then the log itself.
  • Harness verbosity: an explicit --harness-verbosity N beats the count from -v.
  • Stdout: never affected.

Harness verbosity

Harnesses themselves have contollable verbosity levels. Cranking verosity from the harness can be useful for debugging / performance tuning. Thus, add a new --harness-verbosity N option, which is forwarded to harnesses. The result can be very verbose, so it is off by default.
The verbosity level is added to provenance.info.harness_verbosity, so that replaying work orders via bx run gives identical results.


The patch size looks scary but the vast majority of the diff is fairly trivial plumbing.

@ev-br
ev-br force-pushed the logging-phases-1-2 branch from dc0ff95 to 25ab8fa Compare November 20, 2026 12:25
ev-br's Claude Code Agent and others added 12 commits November 20, 2026 14:58
…the plumbing)

bx bench and bx compare now write bx.log in the run's output folder: a header with
the command as given, the config found, each setting with where it came from, and a
footer with the outcome, exit code and elapsed time. The terminal is unchanged at the
default verbosity: what bx prints to stderr goes through the logger as console messages,
and a record that is not one is shown only with -v or -vv.

- new benchx/log.py: the console adapter and handler, the file and --log handlers, one
  line format (UTC to the millisecond, pid, logger, level; continuation lines indented),
  the footer, log.once, log.degraded and log.command
- the switch: --log, --no-log, --log-file PATH (one destination, mutually exclusive),
  `logging` and `log_level` in benchx.conf.json, $BENCHX_LOG_LEVEL and --log-level,
  $BENCHX_VERBOSE with -v and -q; a command without an output folder logs only when a
  flag asks; an unusable log file is refused before anything runs
- cli.main wraps the command: a crash is a console error with its traceback and exit
  status 1, SystemExit passes through, an interrupt or SIGTERM leaves a footer
- the run key and the output folder are settled before any target is resolved, through one
  session.run_directory(); the harness flag check moves into the CLI's check phase, so a
  refusal for a flag still leaves no folder, while a refusal inside the session leaves a
  folder holding only bx.log, which says why
- Config.pick_with_source and the env argument, Config.log_file and Config.own_paths;
  bx config show gets logging, log level and verbosity rows
- adapters/asv.py: its three stderr prints are console messages

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
docs/benchx.md: a paragraph on the log (where it is, its header and footer, the flags, the
environment variables and the config keys, -v and -q), saying plainly that a bx.log records
the command line as given and can contain secrets. docs/design/prototype-design.md: the
output channels, and the logging handlers beside them.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…lan, phase 2)

The log now follows a run. Per order: the protocol, what it records (by name, never a value of
the inherited environment), where each environment name came from, how the subject's identity was
found, the interpreter, and a dirty tree (INFO the first time in a command, DEBUG after, with the
first ten paths git lists). Per case: `starting` before it runs, then one line by its result status
(`error (harness.error): ValueError: boom, exit 0, 1.2 s, load 0.42`, a WARNING; a non-zero exit is a
WARNING too), a files line with absolute paths, and the harness's stdout and stderr (the last 40
lines, DEBUG, or WARNING for a case that did badly). At the end of an order: the results by status.

- runner: refusals through one _refuse(); a rerun of a run key says its result files are overwritten;
  a case loop that stops partway says how many cases were written and that bx ingest stores them;
  an unknown dirty state is a WARNING naming the cause
- session: the run key and folder, each side, each slot before and after it (with the store's outcome),
  the variants' candidate parameters and per-value filters, the baseline's origin, and what the store
  rejected; a round that stops after an earlier slot was ingested says how to compare what is there
- collectors: command, exit, elapsed, a progress line each, the reason for a failure, up to 40 differences
- adapters: run_bounded logs the command, how it ended and its output (a listing's stdout only when it
  failed); run_case logs its command and a missing or unparseable native output; list_cases logs the
  cases; target logs the provider call and the sidecar; bench logs the shorthand's tree
- snapshot: failing git commands, why a tree is dirty, why its state is unknown
- cli: the footer says what an exit-0 command left undone (`1 of 4 cases not measured`); ingest records
- log: harness_output(), and a small buffer so that records made in a command's own checks reach the
  log once it starts

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ging plan, phase 3)

- compare(): a thin wrapper over _compare() that logs the inputs and the outcome from the
  document it returns, so every return path is covered, and the document is unchanged: one record
  per unit with its verdict trail (rounds, reference, effect, noise, k x noise, reason, attempts),
  one per failed invariant (ERROR when it is about the run), the excluded results grouped by reason,
  the missing units, and a summary that is a WARNING when anything was indeterminate, excluded or
  failed. The decisions only _compare() makes are logged where they happen: how --baseline matched, a
  second success of a side in one round overwriting the first, rounds dropped for a success on one side
  only, successes with no observations, and min_rounds below 2
- a read-mode comparison appends its records to the run folder's bx.log when the folder exists, and
  is skipped silently when it cannot be written; --results logs how many documents it ingested, and a
  rejection says the comparison was not run
- snapshot: a working tree identified by a hash of its files (INFO, WARNING when git failed), an
  unreadable file in it, the exclusions behind a tree id, the CMake configuration (WARNING when there is
  no cache or no compiler file), a setup file that is missing or unreadable (naming the variable, never
  its value), the CPU model, the load average and unreadable system facts; the refusal for a checkout that
  cannot be located says why
- store: what a sweep found, every error of a rejected document at DEBUG, an idempotency-conflict with
  both digests, work orders that no result references, a new quantity's unit, the schema promotion by
  field path and type, the time of each read and write, and an absent file read as an empty store
- adapters: pyperf's functions left out without a filter, those with no version, those whose params
  could not be read, a name-only filter match; gbench's partial case; asv's conf that is not JSON
- session, target, cli: how a side was found, the source URI and where it came from, the shorthand's mode,
  the project at INFO, a provider's SOURCE_REF default

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…s for R11 and R22

- store: a write that fails (disk full, permissions) is an ERROR naming the store, the rows that were
  to be added and the partial temporary file; so is a batch that cannot be converted to rows
- session: delivery that fails says where the results are and that bx ingest stores them
- tests: the logger passes only the levels somebody wants (R11); a failed case's `not measured` section
  and `bx compare --tables`, with and without --all, keep stdout free of log lines under --log (R22)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…es, the context, history and validate

- compare: which result supplied each side's declarations, a WARNING naming the fields when the side's
  other results disagree (never their values), and the facts that differ between the sides but that the
  identity policy ignores
- runner: a Google Benchmark context that says CPU frequency scaling is on, or the library is a debug
  build, is a WARNING once per command
- snapshot.ignore: the paths now left out of every tree identity
- cli: `bx history` says when no series matches and warns when a prefix matches several, `bx validate`
  logs its kind and each valid file

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…warning reasons, tables, validation

- asv: each setting of a case's plan with where it came from (the order, sample_time, warmup_time,
  timeout, or the default), the order's min_time when the fixed loops make it useless, a `repeat`
  that is not a form it reads; a `skipped.json`, `facts.json` or worker-environment file that exists
  and cannot be read; a failed module's traceback and the benchmarks the suite switched off with
  skip_benchmark, which asv_runtime now returns in its listing (the worker logs nothing)
- the case line gives the reason behind a quality warning (env-mismatch: how many workers matched;
  env-capture-missing: missing or unparseable; setup-per-call-unbatched: median and minimum) and a
  partial case's completed over attempted; the facts a harness replaces in the context's are named
- tables: what --tables showed and hid, the documents it skipped, and the units the side tables hold that
  the comparison dropped
- core.validation: every schema error at DEBUG with its path and keyword; session: the causes behind the
  conversions that replace them; `bx series` and `bx head` say when nothing matches; an inferred --vary
  says how many results it scanned
- fix: the setup-per-call detail read an observation as a number, and observations may be dicts

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… consults

Every command now logs, at DEBUG, the arguments as argparse understood them (beside the command line as
given, which the header already echoes) and the environment variables benchx reads that decide what it
does: BENCHX_HOME, BENCHX_VERBOSE, BENCHX_LOG_LEVEL, BENCHX_SETUP_FILE, BENCHX_ENV_ALLOWLIST and
BENCHX_SCHEMAS, each with its value or `not set`, and whether PATH is set. These are settings someone
typed for benchx, a fixed list; the rest of the inherited environment is still never dumped. The docs say so.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Right after a case's `starting` record, at DEBUG: its workload and its parameter values (a
parametrized benchmark's own, as asv's repr() strings, and the order's -e variables, which the runner
records as workload parameters too). The listing says once, per parametrized benchmark, its parameters,
their values and how many combinations there are.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…t says

An integer, 0 by default (nothing more than before). Set by `--harness-verbosity N`, else by each `-v`
past `-vv` (`-vvv` is `-vv` and 1, `-vvvv` is `-vv` and 2), else `$BENCHX_HARNESS_VERBOSITY`, else
`adapter.verbosity` in the config file. pyperf and asv are given `--verbose` when it is 1 or more, Google
Benchmark `--v=N`.

- it is an optional work-order field, `harness_verbosity`, so the order records what ran and `bx run`
  repeats it; it is left out of the series identity, and each result's provenance.info records it
- the log keeps up to 2000 lines of each stream of a case's output when it is on, 40 otherwise
- `$BENCHX_VERBOSE` accepts values above 2 for the same rule; `bx config show` has a row
- docs: benchx.md, the work-order schema's table and runner-schema.md

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…a call starts with a line

`bx bench` and `bx run` printed their table, the cases that were not measured,
the result line and the ingest counts with `print`, outside logging. So the log
lacked a round's numbers, and `-q` and `-qq` could not quiet them (a failed case
exits 0 and showed only there).

- `log.console().out(text, tier)`: a record marked `console=True, stdout=True`,
  one record a line. A stdout handler writes it unless `-q` or `-qq` hides it by
  its tier (the table 0, the result line 1, the `not measured` section never);
  the stderr console skips it, `-v` included; the log keeps it at every
  verbosity. A failed write to stdout (a closed pipe) is not swallowed.
- `-q` leaves the `not measured` section and the result line, `-qq` only the
  section; a run that measured everything is silent at `-qq`. When the table is
  hidden and says nothing of a failure, `N of M cases not measured` stands in.
  `bx ingest`, `bx compare`, the queries and `--json` still print as before.
- A call starts with a progress line, `measuring SUITE on TARGET (round R/N,
  slot S)` for `bx bench` and `running the order FILE` for `bx run`, which `-q`
  keeps and `-qq` does not (`console().progress()`: a record's tier lets it past
  the level). It is new on the default stderr.
- The ingest counts after a run are an ordinary console message on stderr, so
  `-q` hides them; `bx ingest` prints its counts on stdout as its answer.
- `--round R/N` says the session has N rounds, numbered 0 to N-1; only the start
  line uses N, and the work order records R alone. `--round R` works as before.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…und R/N

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@ev-br
ev-br force-pushed the logging-phases-1-2 branch from ebf54dc to f871007 Compare November 20, 2026 15:10
@ev-br

ev-br commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator Author

As a specific example, in benchx-dev/benchx_examples#18 applying the diff below to the "workbench" script makes the loop over the rounds mostly quiet and the final bx compare --tables emits asv-style comparisons.
All the gory details from benchx, including per-round timings, are still visible in the log file, bx.log in the out/ directory, but the output is less verbose.

diff --git a/examples/numpy-revisions/compare.sh b/examples/numpy-revisions/compare.sh
index 0b7e643..7992bd3 100755
--- a/examples/numpy-revisions/compare.sh
+++ b/examples/numpy-revisions/compare.sh
@@ -57,7 +57,7 @@ measure() {
     export BUILD PYTHONPATH
     python -c 'import numpy; print(numpy.__version__, numpy.__file__)'   # which numpy this side measures
     bx bench @numpy --adapter asv --suite "$NUMPY/benchmarks/benchmarks" --filter "$filter" \
-        --run-key "$run_key" --round "$round" --slot "$slot"
+        --run-key "$run_key" --round "$round" --slot "$slot" -q
 }
 
 for round in 0 1 2 3 4; do
@@ -65,4 +65,4 @@ for round in 0 1 2 3 4; do
     measure "$contender" "$contender_build" "$round" $((2 * round + 1))
 done
 
-bx compare --run "$run_key" --baseline "$baseline" --tables
+bx compare --run "$run_key" --baseline "$baseline" --tables --all

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add logging

1 participant